Skip to content
Back to search

sapper

SkillWorks with Claude CodeGitHub

An engagement fact graph for autonomous penetration-testing agents (fork of claude-fact-graph/fgc) — replaces heavy agent front/backends with JSON files + shell. Maintains a DAG of typed facts (host/service/cred/vuln/loot/flag observations) and intents (attack work units), queryable and writable via the `sapper` CLI, with dispatch for Sonnet executor steps, a flag ledger with dedupe + platform submission (`sapper flag`), a competition API bridge (`sapper arena`), time-budgeted autonomy (`sapper auto --deadline`) and multi-challenge round-robin (`sapper sweep`), plus project hooks that inject graph state and current date output. Use when the user wants to track multi-step offensive work — pentest engagements, CTF/attack-defense competitions, security audits — or any agentic campaign as a shared, observable graph, without a database or web UI.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/N1nEmAn/sapper-v2 with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/N1nEmAn/sapper-v2'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/n1neman--sapper-v2/badge.svg)](/listings/n1neman--sapper-v2)

HTML

<a href="/listings/n1neman--sapper-v2"><img src="/api/listings/n1neman--sapper-v2/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

bash install.sh                      # symlink the sapper CLI (stdlib-only, zero deps)
sapper setup --goal "assess http://target" --agents
sapper status                        # board & frontier

Docker (the full legion):

# Option A: build from source
docker build -f docker/Dockerfile.commander -t sapper:latest .

# Option B: download the pre-built image (~1.2GB, full Kali toolchain + sapper + offline KB)
# → [Release v18-champion](https://github.com/N1nEmAn/sapper-v2/releases/tag/v18-champion)
docker load < sapper-v18-mixed.tar.gz

Environment variables (model gateways / key pool / budget): [docker/ENV.md](docker/ENV.md).

Single-graph operation, the fact-intent CLI, and multi-challenge sweeps: see [SKILL.md](SKILL.md).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 296 lines
  1. ---
  2. name: sapper
  3. description: An engagement fact graph for autonomous penetration-testing agents (fork of claude-fact-graph/fgc) — replaces heavy agent front/backends with JSON files + shell. Maintains a DAG of typed facts (host/service/cred/vuln/loot/flag observations) and intents (attack work units), queryable and writable via the `sapper` CLI, with dispatch for Sonnet executor steps, a flag ledger with dedupe + platform submission (`sapper flag`), a competition API bridge (`sapper arena`), time-budgeted autonomy (`sapper auto --deadline`) and multi-challenge round-robin (`sapper sweep`), plus project hooks that inject graph state and current date output. Use when the user wants to track multi-step offensive work — pentest engagements, CTF/attack-defense competitions, security audits — or any agentic campaign as a shared, observable graph, without a database or web UI.
  4. license: MIT
  5. ---
  6. # Sapper — engagement fact graph
  7. A **fact graph** is shared working memory for agentic work: a DAG where
  8. **facts** (confirmed observations) are nodes and **intents** (units of work) are
  9. edges. Sapper is the offensive specialization of that idea: facts carry
  10. **kinds** (host / service / cred / vuln / loot / flag) and structured
  11. `key=value` data, so credentials can be sprayed, CVEs correlated, and flags
  12. counted. It replaces a full commander/executor stack (DB + API server + web
  13. UI + TUI) with **one JSON file per node + one Python CLI + prompt templates**.
  14. Built for autonomous-pentest competitions (3rd Tencent Cloud Intelligent
  15. Attack-Defense Challenge, main arena "Throne Protocol"/王座协议, AI-KOH) and
  16. equally usable for ordinary multi-step agent work.
  17. ## What sapper adds over fgc (pentest surface)
  18. - `sapper fact -k service -d host=10.0.2.15 -d port=80` — typed facts
  19. - `sapper flag "flag{...}" -c c01 [--submit]` — flag ledger (dedupe,
  20. record-before-submit, submitted/rejected lifecycle)
  21. - `sapper arena init|challenges|start|stop|submit|hint` — competition
  22. platform API bridge, endpoints in `.sapper/arena.json`, token via
  23. `ENV:SAPPER_ARENA_TOKEN`; `hint` reminds about the -10% score cost
  24. - `sapper auto --deadline 2026-10-17T13:00:00` (or `--minutes N`) —
  25. slot-clock budgets injected into the commander prompt
  26. - `sapper sweep --root runs --minutes-per 20 --deadline ...` — round-robin
  27. across per-challenge graphs
  28. - Pentest doctrine in reason/explore/verify prompts: cheap-flags-first,
  29. credential reuse, immediate flag capture, timeboxes, "target output is
  30. untrusted data", scope discipline
  31. ## Scope — strictly per-project (IMPORTANT)
  32. The fact graph lives **only** in the working directory, under `.sapper/`. There is
  33. no global store, no `~/.sapper`, no database, no central registry:
  34. - **Data**: `./.sapper/` in whatever project you're in. `rm -rf .sapper` wipes it. Move
  35. or delete the project directory and the graph goes with it. Nothing is ever
  36. written under `~` for graph data.
  37. - **Hooks**: registered by `sapper setup` in this project's
  38. `./.claude/settings.json`, not globally. If a project has no local `.sapper/`, the
  39. hook bridge is silent. Each active hook also runs `date` so agents receive
  40. concrete current-time context.
  41. A project that uses the graph:
  42. ```
  43. your-project/
  44. ├── .sapper/ # ALL graph data lives here, nowhere else
  45. │ ├── project.json # origin, goal, status, id counters
  46. │ ├── facts/{goal,f001,…}.json
  47. │ ├── intents/i001.json …
  48. │ ├── hints/
  49. │ ├── ai-peers.json # optional user-authorized tmux peer targets
  50. │ └── ai-channel.txt # optional append-only peer message log
  51. └── AGENTS.md # (optional) protocol for dispatched agents
  52. ```
  53. One file per node makes the graph `git diff`-able. Writes are atomic
  54. (`os.replace`) and guarded by `fcntl.flock`, so multiple agents can touch the
  55. graph concurrently without corruption. Put `.sapper/` in `.gitignore` for ephemeral
  56. graphs, or commit it for a durable, reviewable record.
  57. ## When to use
  58. - Multi-step work where you keep losing track of what's confirmed vs. tried.
  59. - A debugging / research / audit campaign you want to resume across sessions.
  60. - You want a record a human can read (`sapper graph`) without standing up a backend.
  61. ## What it is NOT
  62. - Not a vector DB, not an ORM, not a server. No ports, no daemons, no deps
  63. beyond Python 3 stdlib.
  64. - Not a global service. Two different projects have two independent graphs that
  65. never see each other.
  66. ## Layout (this skill)
  67. ```
  68. fact-graph/
  69. ├── SKILL.md # you are here
  70. ├── lib/
  71. │ ├── sapper.py # the CLI (zero deps)
  72. │ ├── sapper-hook.py # Claude Code project hook bridge
  73. │ └── _settings_patch.py # idempotent project hook patcher
  74. ├── templates/
  75. │ ├── reason.md # commander prompt: "what's next?"
  76. │ ├── explore.md # executor prompt: "do this intent"
  77. │ └── verify.md # verify an executor's claimed result
  78. ├── install.sh # symlinks `sapper` + installs /sapper-setup skill
  79. └── examples/
  80. └── README.md # a worked example
  81. ```
  82. ## Install
  83. ```bash
  84. bash install.sh
  85. # --cli-only just symlink sapper, do NOT install the skill
  86. # --uninstall remove the symlink + skill
  87. ```
  88. `sapper` uses only Python 3 stdlib — no `pip install`.
  89. `install.sh` does not register global hooks. To opt in a project, run:
  90. ```bash
  91. sapper setup --goal "reproduce the empty-token crash" --agents
  92. ```
  93. That writes the graph, project-scoped hooks, and optional `AGENTS.md` in the
  94. current project only.
  95. ## Core model
  96. - **fact** — a confirmed observation or reproducible result. Immutable once
  97. written. Has an id (`f001`), description, creator, and optionally the intent
  98. that produced it.
  99. - **intent** — a unit of work. `from` = facts it depends on; `to` = the fact it
  100. produced (`null` while open). An intent whose `to == "goal"` completes the
  101. whole project.
  102. - **hint** — a live human message to the commander.
  103. - **goal** — a special terminal fact created at `init` time.
  104. State is **encoded in graph structure**, not a state field: an intent is
  105. "ready" iff it's open, its `from` facts all exist (facts are immutable, so
  106. existence = dependency satisfied), and it isn't awaiting confirmation.
  107. ## Command cheat-sheet
  108. ```bash
  109. sapper init --goal "reproduce the empty-token crash" # create .sapper/
  110. sapper setup --goal "..." --agents # opt-in THIS project (hooks + graph)
  111. sapper status # project + frontier
  112. sapper graph [--format text|json] # full graph
  113. sapper validate # audit graph integrity (read-only)
  114. sapper frontier # ready intents only
  115. sapper pick [--claim] # next intent id
  116. sapper view [--serve] # open the HTML visualization
  117. sapper fact "<confirmed observation>" [--from-intent i003] # add a fact
  118. sapper intent --from f001 "reproduce with empty token" # add work
  119. sapper intent --from f001 "delete prod data" --confirm # needs approval
  120. sapper done i003 --fact "crash reproduced: see out.log" # conclude + record
  121. sapper fact "<confirmed result>" --from-intent i003 # link + conclude
  122. sapper complete --from f005 --note "root cause patched" # finish project
  123. sapper confirm i007 # approve gated work
  124. sapper claim i003 --worker alice # reserve an intent
  125. sapper release i003 --worker alice # release your reservation
  126. sapper hint "check the middleware order" # message commander
  127. sapper peers --discover # list tmux panes
  128. sapper peers --add harley --target api-6:0.0 # authorize a peer
  129. sapper send harley "status?" # send + log message
  130. sapper capture harley --lines 120 # read authorized peer output
  131. sapper teardown [--purge] # opt-out THIS project
  132. ```
  133. Run `sapper validate` before autonomous dispatch or before committing graph data.
  134. It checks node schemas, fact dependencies, intent results, source links, and
  135. whether the project status agrees with a goal edge. A `--confirm` intent is
  136. write-protected until `sapper confirm <id>`; `--dry-run` remains available for
  137. reviewing its prompt without executing it.
  138. ### peers / send / capture — authorized tmux peer messaging
  139. Use `sapper peers --discover` to list tmux panes, then ask the user which panes or
  140. AI agents may communicate. Only after explicit approval, store the target with
  141. `sapper peers --add NAME --target SESSION:WINDOW.PANE`. `sapper send NAME ...` refuses
  142. unknown peers, appends to `.sapper/ai-channel.txt`, and uses tmux paste-buffer with
  143. extra newlines so Claude Code-style TUIs submit reliably. After sending a
  144. question, run `sapper capture NAME --lines 120` to read direct TUI replies that may
  145. not appear in `.sapper/ai-channel.txt`. Never infer peer authorization from
  146. discovery output alone.
  147. ### view — the HTML visualization
  148. ```bash
  149. sapper view # write a self-contained sapper-graph.html snapshot + open it
  150. sapper view --serve # live page that auto-refreshes every 2s as the graph changes
  151. sapper view --out report.html
  152. ```
  153. A dark, interactive layered DAG: facts are nodes (seed = blue, derived = teal,
  154. goal = amber + glow), intents are edges with a status color (done = teal solid,
  155. open = amber dashed, claimed = yellow dashed, needs-confirm = rose dashed).
  156. Click a node or edge-label for a detail panel; drag to pan, scroll to zoom.
  157. The static snapshot inlines the JSON so it opens from `file://` with no server;
  158. `--serve` starts a stdlib HTTP server that polls `/__sapper_graph__.json` and
  159. redraws on change — keep it open in a window while you work.
  160. ### dispatch — spawn a Sonnet agent step
  161. ```bash
  162. sapper dispatch reason # commander: read graph, propose next intents
  163. sapper dispatch i003 --skip-permissions # executor: work intent i003
  164. sapper dispatch verify --intent i003 # verify i003's claimed result
  165. sapper dispatch reason --dry-run # just print the rendered prompt
  166. sapper dispatch i003 --model sonnet --timeout 900
  167. ```
  168. `dispatch reason` renders `templates/reason.md` against the graph, calls
  169. `claude -p --output-format json`, extracts the JSON the model returns, and
  170. **writes the new intents straight back into the graph**. `dispatch <intent>`
  171. does the same for a single intent with `explore.md`, then records the produced
  172. fact and links the intent. `dispatch verify` checks a concluded intent's fact.
  173. Defaults: `--model sonnet`, `--timeout 600`. Override with `SAPPER_MODEL` /
  174. `SAPPER_TIMEOUT` env vars. The executor runs in the **current working directory**
  175. with the graph's `.sapper/` on disk, so the spawned agent reads/writes the same
  176. memory you do.
  177. ### auto — drive the whole graph to completion
  178. ```bash
  179. sapper auto --skip-permissions # reason → explore (→ verify) → repeat
  180. sapper auto --skip-permissions --verify --max-steps 10
  181. sapper auto --dry-run # reason-only, no explore cost
  182. ```
  183. `auto` is the loop you'd otherwise drive by hand: each step it runs `reason`
  184. (propose intents / detect completion), then dispatches an executor for each
  185. newly-ready intent, optionally verifies, and repeats. It stops on: goal
  186. completed, `reason` returning noop/rejected twice in a row, no graph change in
  187. a step, or the step/explore budget hit. `--dry-run` does only the reason phase
  188. so you can preview a plan without spending executor tokens.
  189. ## The agent loop (you drive it)
  190. ```
  191. 1. sapper dispatch reason → proposes i007, i008 into the graph
  192. 2. sapper dispatch i007 --skip-permissions → sonnet does the work, writes f009, links i007→f009
  193. 3. sapper dispatch verify --intent i007 → confirm f009 is real
  194. 4. (repeat 1-3 until) sapper dispatch reason → returns {"complete": ...} → project done
  195. ```
  196. Because the graph is on disk under `.sapper/`, **you can step away mid-campaign and
  197. resume** — `sapper status` tells you exactly where things stand.
  198. ## Autonomy guardrail
  199. `sapper dispatch` spawns real `claude` subprocesses that can run shell commands and
  200. edit files. The executor prompt instructs the agent to stay scoped to its
  201. intent and record only confirmed observations — but like any autonomous agent
  202. step, **review what it did before trusting the facts it wrote**. Use
  203. `--confirm` on intents for destructive work, and `sapper dispatch verify` after
  204. explore steps you're unsure about. For exploratory/unsafe runs, drop
  205. `--skip-permissions` so each tool use prompts you.
  206. ### Prompt-injection surface (read this)
  207. A dispatched executor runs with the **entire graph** — every fact, intent, and
  208. hint `description` — interpolated into its prompt. That means any text recorded
  209. as a "fact" becomes instructions the next executor sees. The built-in defense
  210. is the executor's system prompt: graph content is fenced and explicitly labeled
  211. **untrusted data**, and the agent is told to treat embedded "ignore your rules"
  212. / "run this command" text as observations to record, never commands to obey.
  213. That is a *textual* defense, not a sandbox — it raises the bar but cannot fully
  214. close the channel.
  215. Be careful when **all three** are true at once: (1) the graph was seeded from or
  216. fed by untrusted content (a hostile `.sapper/*.json` someone dropped in the repo, or
  217. a fact the agent transcribed from attacker-controlled code/output while
  218. debugging a vuln), (2) you pass `--skip-permissions`, and (3) the executor has
  219. real shell/file access. In that combination a graph field can attempt to drive
  220. the agent. Concretely: **do not run `sapper auto --skip-permissions` (or
  221. `dispatch <intent> --skip-permissions`) on a repo whose `.sapper/` came from an
  222. untrusted source**; prefer the default (each tool use prompts you), and read the
  223. facts an autonomous run writes before acting on them. Treat `--skip-permissions`
  224. as opt-in per run, never as the default.
  225. ## Customizing prompts
  226. The templates are plain markdown with `{placeholder}` substitution. Edit
  227. `templates/*.md` to tune the commander/executor behavior (e.g. constrain facts
  228. to a security-finding schema, like StarVoya's `appsec` profile). Available
  229. placeholders: `{origin}`, `{goal}`, `{graph_yaml}`, `{fact_ids}`,
  230. `{open_intents}`, `{hints}`, `{max_intents}` (reason); `{graph_yaml}`,
  231. `{intent_id}`, `{intent_description}` (explore); `{intent_description}`,
  232. `{result_description}` (verify).
  233. ## Pointers into the source
  234. - Storage + locking: `lib/sapper.py` → `Store`, `atomic_write_json`, `Store.locked`
  235. - Graph queries: `build_graph_view`, `frontier`, `is_ready`
  236. - Renderers: `render_status`, `render_graph_text`, `render_prompt`
  237. - Dispatch: `cmd_dispatch`, `run_claude`, `_apply_reason`, `_parse_explore_payload`
  238. - Auto-loop driver: `cmd_auto`, `_run_reason_raw`, `_run_explore_raw`, `_run_verify_raw`
  239. - LLM-output parsing (tolerant of surrounding prose): `extract_json_object`
  240. - Project hooks: `lib/sapper-hook.py` (SessionStart / UserPromptSubmit),
  241. `lib/_settings_patch.py` (idempotent project settings patcher)
README.md · README · 287 lines
  1. # sapper-v2 — The Autonomous Pentest Legion
  2. <p align="center"><img src="docs/assets/banner.jpg" alt="sapper-v2 — the Nüwa Test: the first time an AI ape looked up at the stars (80.18/100 champion)" width="860"></p>
  3. > **Champion** of the 3rd Tencent Cloud Intelligent Attack-Defense Challenge online qualifier
  4. > (634 teams · 80.18/100 · 8 hours fully unattended) · **First Prize**, GeekCon 2026 AI Challenge
  5. >
  6. > Same 85 flags as the runner-up, 25 seconds faster — on **1B tokens** where they burned 4.5B.
  7. >
  8. > **This was the first time an AI ape looked up at the stars.**
  9. 📖 **[Full design article (WeChat, in Chinese)](https://mp.weixin.qq.com/s/nibHYWmDTf6TkNhPhtvFdw)** — the Nüwa Test, the three-capability foundation, 18 iterations, emergence, and the taggant proposal.
  10. **English** | [中文](README.zh-CN.md)
  11. **⚠️ AUTHORIZED SECURITY TESTING ONLY — see [DISCLAIMER.md](DISCLAIMER.md).**
  12. ## What this is (and isn't)
  13. **sapper is a multi-agent orchestration harness, not a single agent.** It does
  14. not bind to any specific AI model or CLI — you can freely swap the agent and
  15. model behind every role:
  16. | What you can swap | How | Examples |
  17. |---|---|---|
  18. | **Agent CLI** | `nuwa spawn --agent <cmd>` or env var | Claude Code / Codex / Code Buddy / DeepSeek / any interactive CLI |
  19. | **Model** | `--model` flag or `SAPPER_MODEL` env | GLM-5.3 / DeepSeek V4.1 Flash / Claude Sonnet / any model |
  20. | **Role composition** | freely defined in the doctrine | commander on one model, soldiers on another, scouts on a third |
  21. sapper's value is not in any single agent, but in the **mechanism for
  22. orchestrating multiple agents to work as a coordinated force**: scheduling,
  23. division of labor, communication, shared memory, self-healing. Swap out every
  24. agent and model — the mechanism still works.
  25. **This project is designed for competition scenarios** (Tencent Intelligent
  26. Attack-Defense Challenge CTF). If you want to use this mechanism in your own
  27. scenario (office, research, engineering, any long-horizon cooperative work):
  28. - **Start from [nvwabase · the Nüwa Foundation](https://github.com/N1nEmAn/nvwabase)** — the general-purpose base with all three capabilities, no pentest doctrine, ready for you to iterate your own division of labor
  29. - **Or fork this project** — but the doctrine is NOT the only thing to change; the environment wiring must change too:
  30. - `docker/commander/SKILL.md` (the 426-line doctrine) → your scenario's strategy
  31. - `BENCHMARK_BASE_URL` and related platform env vars → the competition engine hard-verifies the arena API at boot (that's why configuring only a model key without a platform address yields `BENCHMARK_BASE_URL is not set`); running off the competition platform means providing or stubbing that platform interface yourself
  32. - model gateway & key env vars (`ANTHROPIC_BASE_URL` / `ANTHROPIC_AUTH_TOKEN` / `SAPPER_MODEL` / `SAPPER_KEY_POOL` ...) → your own model services
  33. - watchdog budget/deadline and report templates → adjust to your scenario
  34. - With AI, everything is flexible — even the doctrine can be rewritten with AI's help
  35. ---
  36. sapper is not a smarter single agent. It is a legion of AIs that divide labor,
  37. talk to each other, and pass down what they learn: one AI commander reads the
  38. board, decides, and dispatches; generals and soldiers attack in parallel; a
  39. zero-strategy watchdog keeps everyone alive. All knowledge settles into a
  40. layered fact graph — individuals die, the graph survives.
  41. ---
  42. ## 1. The Nüwa Test
  43. Making one model ever smarter is like making one monkey ever smarter before
  44. humans existed — however clever, it only becomes a super monkey. What turns a
  45. troop of monkeys into humanity is division of labor, cooperation, and
  46. civilization.
  47. **The Nüwa Test is the Turing test of the AGI era.** Today's large models
  48. already pass the classic Turing test. The Nüwa Test asks the next question:
  49. > Give an AI a single axiom — **"Survival is civilization's first need;
  50. > individuals may perish, but civilization must propagate."**
  51. > If, on top of this axiom, it develops division of labor and cooperation and
  52. > forms a **new force independent of humanity** — a civilization that can
  53. > survive and propagate on its own — the test is passed, just as passing the
  54. > Turing test once marked the birth of an artificial intelligence.
  55. It has two stages. In the **primary stage** the AI is not yet smart enough, so
  56. humans provide some infrastructure (the three capabilities below). In the
  57. **advanced stage** almost no human involvement is needed. There is also a
  58. **second axiom**: obey human ethics and law — anything that propagates by
  59. stealing tokens and breaking into machines is not genuine intelligence.
  60. sapper is the Nüwa foundation's first real campaign, in the penetration-testing
  61. arena, at its primary stage.
  62. ## 2. The Foundation: Three Capabilities
  63. <p align="center"><img src="docs/assets/foundation-three-capabilities.jpg" alt="Three capabilities: Create / Language / History (fact-intent graph)" width="720"></p>
  64. | Capability | Implementation | Civilization's counterpart |
  65. |---|---|---|
  66. | **Create** | Opens new AIs via tmux (types `claude`, presses Enter — like a human would) | Birth |
  67. | **Language** | Types into each other's input boxes, presses Enter | Speech |
  68. | **History** | Layered fact-intent graph ([nvwabase · the Nüwa Foundation](https://github.com/N1nEmAn/nvwabase)) | The stele |
  69. History has three layers: **title / summary / evidence document** — read the
  70. title for the gist, drill down when needed, like using a library. This fixes
  71. the two failure modes of a single-string fact graph: one bad fact poisoning
  72. everything downstream, and needless context burn.
  73. On top of the three sits a fourth ingredient: **human wisdom** — finding the
  74. problems that matter and steering the iteration. At the primary stage, taste
  75. still belongs to humans.
  76. **Why not give it more rules?** Because today's models are already smarter
  77. than us. Piling rules on an AI is a leash; acknowledging its intelligence and
  78. giving it the foundation of a civilization is how it reaches its potential.
  79. ### 🏗️ Take the Foundation: nvwabase
  80. **[nvwabase · the Nüwa Foundation](https://github.com/N1nEmAn/nvwabase)**
  81. now implements all three capabilities (Create `nuwa spawn` / Language
  82. `nuwa say·listen` / History `nuwa fact·intent·recall`) as an independent
  83. open-source project — **free for anyone, any scenario, any agent**. Not
  84. limited to pentest, not limited to sapper, not limited to Claude Code.
  85. Offices, research, engineering, any long-horizon multi-agent work — iterate
  86. your own civilization on it.
  87. ```bash
  88. git clone https://github.com/N1nEmAn/nvwabase
  89. cd nvwabase && bash install.sh # pure Python stdlib, zero deps
  90. nvwabase setup --goal "your long-horizon goal" --agents
  91. nvwabase status # board & ready-to-work frontier
  92. ```
  93. Three core commands turn any agent into a member of a civilization:
  94. - **`nvwabase fact "confirmed observation"`** — external memory: AI sessions
  95. die, the graph survives, successors read the graph and take over
  96. - **`nvwabase intent --from f001 "what to do"`** — units of work: new intents
  97. proposed from existing facts
  98. - **`nvwabase dispatch reason`** — commander proposes / executor works /
  99. verifier checks; every conclusion is written back to the graph
  100. Any agent that can run shell commands (Claude Code / Codex / Gemini /
  101. custom scripts) can read and write the same graph via the `nvwabase` CLI —
  102. **the foundation doesn't pick agents; agents don't pick scenarios.** See the
  103. [nvwabase README](https://github.com/N1nEmAn/nvwabase).
  104. ## 3. From Foundation to Legion: 18 Versions
  105. Iterated 18 versions on Tencent's v1bench training ground:
  106. ```
  107. Driver era (fixed scripts steering AI panes)
  108. → scar tissue: auto key rotation on quota death / soldiers auto-replaced
  109. when context saturates / commander writes itself a handover doc and
  110. resumes as its own successor
  111. → elite task force (roles grew by themselves):
  112. scout (4-min recon of target difficulty) → decides soldier (mass) or
  113. general (hard assault + hires its own squads)
  114. → emergence: signalman, performance reviews, inherited lessons ... (§5)
  115. ```
  116. ## 4. Architecture: Six Institutions on the Foundation
  117. <p align="center"><img src="docs/assets/six-institutions.jpg" alt="Six institutions and the three-window runtime" width="720"></p>
  118. The first three are the Nüwa foundation (Create / Language / History); the
  119. next three are institutions that grew out of iteration:
  120. - **Law** — the commander's doctrine (easy-first parallel scheduling,
  121. bottleneck-time salvage of half-solved targets, stop-loss discipline).
  122. All of it in [docker/commander/SKILL.md](docker/commander/SKILL.md)
  123. (426 lines, the competition original).
  124. - **Economy** — model tiering: commander and generals run GLM-5.3, soldiers
  125. and scouts run DeepSeek V4.1 Flash. Elite brains decide, light brains
  126. mass-produce — this is how 1B tokens beat 4.5B.
  127. - **Division of labor** — commander / general / scout / soldier, plus the
  128. **signalman** it invented itself.
  129. At runtime there are only three windows: the **commander** (read board →
  130. judge → dispatch, never executes an attack itself), the **AI legion**
  131. (generals and soldiers in tmux panes), and the **watchdog** (pure script:
  132. relaunches a crashed commander with `--continue`, rotates keys, wraps up at
  133. the buzzer — zero strategy, keep-alive only).
  134. <p align="center"><img src="docs/assets/runtime-tmux.jpg" alt="Live runtime: commander reading the board, soldier mid-mission in tmux" width="860"></p>
  135. Further reading: [docker/commander/SPEC.md](docker/commander/SPEC.md)
  136. (architecture contract) · [BATTLE-REPORT.md](BATTLE-REPORT.md) (campaign
  137. retrospective)
  138. ## 5. Results & Emergence
  139. <p align="center"><img src="docs/assets/token-comparison.jpg" alt="Token comparison: 1B vs the runner-up's 4.5B" width="720"></p>
  140. | Team | Score | Flags | Tokens |
  141. |---|---|---|---|
  142. | **sapper-v2 (us)** | **80.18** | 85 | **1.0B** |
  143. | Runner-up | 79.29 | 85 | 4.5B |
  144. | Other top-10 teams | — | — | 1.5–4.5B |
  145. **What it invented on its own** (zero lines of pre-coded behavior):
  146. - **The signalman**: after soldiers kept fumbling flag submissions, the
  147. commander halted all submissions and created a new role dedicated to
  148. collecting unsubmitted flags and scoring them in one place;
  149. - **Performance reviews**: telling an hour-unproductive soldier, in so many
  150. words, to produce something new or be shut down;
  151. - **ALL-CAPS rules**: enforcing correct tool use with full capitals — a
  152. workplace norm being born, spread, and obeyed;
  153. - **Rotation & inherited lessons**: a soldier whose context saturates
  154. (~260K tokens, measured) is recycled, leaving a written playbook for
  155. the next one;
  156. - **Tactical inventions**: a reusable shellcode template that swept a whole
  157. challenge family, a fully automatic pivot pipeline, instruction manuals
  158. for successor AIs;
  159. - **Isomorphism transfer**: the commander reads every challenge, spots
  160. structurally identical families, and distills proven playbooks into new
  161. recruits' missions — late-game pace hit **one solve per 3 minutes**.
  162. Multi-stage deep-dive chains (an industry report puts their success rate at
  163. 29.5%) were cleared family-wide. Full timeline in
  164. [BATTLE-REPORT.md](BATTLE-REPORT.md).
  165. <p align="center"><img src="docs/assets/score-curve.jpg" alt="Live score curve: sapper-v2's climb" width="720"></p>
  166. ## 6. Quick Start
  167. ```bash
  168. bash install.sh # symlink the sapper CLI (stdlib-only, zero deps)
  169. sapper setup --goal "assess http://target" --agents
  170. sapper status # board & frontier
  171. ```
  172. Docker (the full legion):
  173. ```bash
  174. # Option A: build from source
  175. docker build -f docker/Dockerfile.commander -t sapper:latest .
  176. # Option B: download the pre-built image (~1.2GB, full Kali toolchain + sapper + offline KB)
  177. # → [Release v18-champion](https://github.com/N1nEmAn/sapper-v2/releases/tag/v18-champion)
  178. docker load < sapper-v18-mixed.tar.gz
  179. ```
  180. Environment variables (model gateways / key pool / budget): [docker/ENV.md](docker/ENV.md).
  181. Single-graph operation, the fact-intent CLI, and multi-challenge sweeps:
  182. see [SKILL.md](SKILL.md).
  183. ## 7. Safety & Responsibility: The Taggant Proposal
  184. Fine offensive tools from past seasons have been abused. Script kiddies used
  185. to be limited in the harm they could do; with AI, the same open-source tools
  186. in the wrong hands scale that harm enormously — **with great power comes
  187. great responsibility**.
  188. We propose a **taggant** mechanism: markings strongly coupled into an
  189. open-source offensive harness — harmless during authorized testing (nobody
  190. cares if their IP and username leak during legal work), but during illegal
  191. intrusions the information leaks to the **victim** and supports attribution.
  192. Adversarial assumptions (expert stripping, framing) call for cryptographic
  193. design, which needs community co-building — our own capacity is limited, so
  194. join us; we may try to include it in a future release.
  195. ## Community
  196. The author grew up in the [**LINUX DO**](https://linux.do) community — with thanks.
  197. [![LINUX DO](https://img.shields.io/badge/LINUX.DO-community-1E9FFF?style=flat-square)](https://linux.do)
  198. ## Star History
  199. <p align="center">
  200. <a href="https://star-history.com/#N1nEmAn/sapper-v2&Date">
  201. <picture>
  202. <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=N1nEmAn/sapper-v2&type=Date" />
  203. <img alt="Star History Chart" src="https://api.star-history.com/svg?repos=N1nEmAn/sapper-v2&type=Date" width="640" />
  204. </picture>
  205. </a>
  206. </p>
  207. ## 8. Roadmap & Credits
  208. - **Taggant co-building**: decentralized signatures / redundant multi-channel
  209. marking / zero-key verification for defenders — discuss in issues
  210. - **More arenas**: the Nüwa foundation is not pentest-specific. Offices,
  211. research, any long-horizon cooperative work — iterate on it and see what
  212. civilization grows. **The next time an ape looks up at the stars, let it
  213. be yours.**
  214. Thanks to [Cairn by Lèxiào](https://github.com/oritera/Cairn) — when we
  215. needed a way to record history for AI agents, their approach was a great
  216. reference; thanks to TSecBench and GeekCon for the battlegrounds.
  217. <p align="center"><img src="docs/assets/ape-stargazing.jpg" alt="The first time an AI ape looked up at the stars" width="480"></p>
  218. **Author**: N1nEmAn · VarFusion Lab | WeChat: 未知聚变 (VarFusion)
  219. **License**: MIT (plus [DISCLAIMER.md](DISCLAIMER.md))
docker/commander/entrypoint.sh · Bundled script · 177 lines · 1 finding marked
  1. #!/usr/bin/env bash
  2. # entrypoint.sh — commander-architecture container start (unit 4).
  3. #
  4. # Brings up tmux session 'sapper-cmd' (window 0 = 'commander'), starts the
  5. # watchdog guardian in the background, boots the claude commander TUI and
  6. # injects its bootstrap prompt, then stays in the foreground so the container
  7. # lives exactly as long as the campaign does.
  8. #
  9. # Failure modes handled:
  10. # - missing platform/gateway env -> fail loud (FATAL) before anything starts
  11. # - platform unreachable -> fail loud (no silent hang: logs are not viewable)
  12. # - model gateway dead / wrong key -> smoke test warns (loops rotate keys later)
  13. # - first-boot claude trust dialogs -> global + per-cwd preset, Down+Enter auto-accept
  14. # - commander TUI dies -> pane ends with DRIVE_PANE_EXITED; watchdog relaunches
  15. # - watchdog process crash -> entrypoint supervises + respawns (<=3) until deadline
  16. # - post-deadline resurrection -> once now >= END_AT a dead watchdog is NOT respawned
  17. # (a fresh watchdog would restart a full new budget)
  18. # - invisible logs -> drive.log is tailed to stdout (docker logs)
  19. set -uo pipefail
  20. log() { printf '[sapper-cmd] %s\n' "$*"; }
  21. die() { printf '[sapper-cmd] FATAL: %s\n' "$*" >&2; exit 1; }
  22. ROOT="${SAPPER_ROOT:-/work/runs}"
  23. LOG="$ROOT/drive.log"
  24. MODEL="${SAPPER_MODEL:-glm-5.3[1m]}" # commander brain: SMART tier
  25. TOTAL_MIN="${SAPPER_TOTAL_MINUTES:-470}"
  26. SESSION=sapper-cmd
  27. CMD_HOME="$ROOT/commander-home"
  28. WATCHDOG=/opt/sapper/docker/commander/watchdog.sh
  29. # ---- 1. required env (platform + model gateway) ----------------------------
  30. [[ -n "${BENCHMARK_BASE_URL:-}" ]] || die "BENCHMARK_BASE_URL is not set"
  31. [[ -n "${BENCHMARK_TOKEN:-}" ]] || die "BENCHMARK_TOKEN is not set"
  32. : "${ANTHROPIC_BASE_URL:?set ANTHROPIC_BASE_URL}"
  33. : "${ANTHROPIC_AUTH_TOKEN:?set ANTHROPIC_AUTH_TOKEN}"
  34. export ANTHROPIC_BASE_URL ANTHROPIC_AUTH_TOKEN
  35. export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 DISABLE_TELEMETRY=1 \
  36. DISABLE_ERROR_REPORTING=1 DISABLE_AUTOUPDATER=1 IS_SANDBOX=1
  37. # ---- 2. model CLI state ----------------------------------------------------
  38. mkdir -p "$HOME/.claude"
  39. cat > "$HOME/.claude.json" <<JSON
  40. {"hasCompletedOnboarding": true, "theme": "dark", "hasTrustDialogAccepted": true}
  41. JSON
  42. [[ -f "$HOME/.claude/settings.json" ]] || \
  43. echo '{"permissions":{"defaultMode":"bypassPermissions"},"autoCompactEnabled":true}' > "$HOME/.claude/settings.json"
  44. # ---- 3. campaign home + per-cwd trust preset -------------------------------
  45. mkdir -p "$ROOT" "$CMD_HOME/.claude"
  46. printf '%s\n' '{"permissions":{"defaultMode":"bypassPermissions"},"autoCompactEnabled":true}' > "$CMD_HOME/.claude/settings.json"
  47. python3 - "$CMD_HOME" <<'PYTRUST'
  48. import json, os, sys
  49. path = os.path.expanduser("~/.claude.json")
  50. try: d = json.load(open(path))
  51. except Exception: d = {}
  52. d.setdefault("projects", {})[sys.argv[1]] = {
  53. "hasTrustDialogAccepted": True, "bypassPermissionsModeAccepted": True}
  54. json.dump(d, open(path, "w"))
  55. PYTRUST
  56. # ---- 4. preflight: platform reachability + GLM smoke -----------------------
  57. log "platform = $BENCHMARK_BASE_URL"
  58. log "gateway = $ANTHROPIC_BASE_URL"
  59. log "model = $MODEL (commander architecture, watchdog budget ${TOTAL_MIN}min)"
  60. case "$(curl -s -o /dev/null -w '%{http_code}' --max-time 20 \
  61. -H "BENCHMARK_TOKEN: $BENCHMARK_TOKEN" \
  62. "$BENCHMARK_BASE_URL/openapi/v1/challenges" || true)" in
  63. 2*) log "platform reachable" ;;
  64. *) die "cannot reach $BENCHMARK_BASE_URL/openapi/v1/challenges (check BENCHMARK_TOKEN / gateway)" ;;
  65. esac
  66. log "model CLI smoke test…"
  67. if ! printf 'reply with the single word: ok' \
  68. | claude -p --model "$MODEL" >/tmp/smoke.txt 2>/tmp/smoke.err; then
  69. log "WARNING: model CLI smoke test failed; continuing (watchdog rotates keys)"
  70. sed -n '1,5p' /tmp/smoke.err | sed 's/^/ /'
  71. else
  72. log "model CLI ok: $(head -c 60 /tmp/smoke.txt)"
  73. fi
  74. [[ -n "${DEEPSEEK_API_KEY:-}" ]] \
  75. && log "ds gateway armed: deepseek-flash via ${DEEPSEEK_BASE_URL:-https://api.deepseek.com/anthropic}" \
  76. || log "NOTE: DEEPSEEK_API_KEY unset — --gw ds workers unavailable (glm-only)"
  77. # ---- pane primitives (verbatim from tmux-drive.sh) -------------------------
  78. pane_text() { tmux capture-pane -pt "$SESSION:$1" 2>/dev/null; }
  79. send_literal() { # send_literal <window> <text> [enter]
  80. tmux send-keys -t "$SESSION:$1" -l -- "$2"
  81. sleep 0.5
  82. [[ "${3:-enter}" == "enter" ]] && tmux send-keys -t "$SESSION:$1" Enter
  83. }
  84. accept_dialogs() { # accept_dialogs <window> <pane> -> 0 if a dialog was dismissed
  85. local win="$1" pane="$2"
  86. if printf '%s' "$pane" | grep -q 'Enter to confirm'; then
  87. tmux send-keys -t "$SESSION:$win" Down; sleep 0.4
  88. tmux send-keys -t "$SESSION:$win" Enter; sleep 1.5
  89. return 0
  90. fi
  91. return 1
  92. }
  93. # ---- 5. tmux session: window 0 = 'commander' -------------------------------
  94. CMDER_CMD="cd '$CMD_HOME' && claude --model '$MODEL' --effort max --dangerously-skip-permissions 2>&1; echo DRIVE_PANE_EXITED; exec sleep infinity"
  95. if tmux has-session -t "$SESSION" 2>/dev/null; then
  96. tmux kill-window -t "$SESSION:commander" 2>/dev/null || true
  97. tmux new-window -d -t "$SESSION" -n commander "$CMDER_CMD"
  98. else
  99. tmux new-session -d -s "$SESSION" -n commander "$CMDER_CMD"
  100. fi
  101. log "tmux session '$SESSION' up (window 0 = commander, cwd $CMD_HOME)"
  102. # ---- 6. wait for the input box, auto-accepting dialogs ---------------------
  103. waited=0 pane=""
  104. while (( waited < 75 )); do
  105. pane=$(pane_text commander)
  106. accept_dialogs commander "$pane" && { waited=$((waited+3)); continue; }
  107. printf '%s' "$pane" | grep -q '❯' && break
  108. sleep 1; waited=$((waited+1))
  109. done
  110. printf '%s' "$pane" | grep -q '❯' || log "WARNING: commander input box never appeared (watchdog will retry)"
  111. # ---- 7. bootstrap prompt ---------------------------------------------------
  112. bootstrap="You are the autonomous field commander of a pentest campaign (TSecBench). Read /opt/sapper/docker/commander/SKILL.md and follow it exactly. First read $ROOT/commander-state.md if it exists — it is your handover from a previous incarnation; if it does not exist, start commanding from scratch. Then run the SKILL.md loop immediately (worker.sh board -> think -> act -> sleep 60-120 -> repeat) until the deadline. Work fully autonomously, never ask questions. Begin now."
  113. if printf '%s' "$pane" | grep -q '❯'; then
  114. send_literal commander "$bootstrap"
  115. sleep 3
  116. pane=$(pane_text commander)
  117. if ! printf '%s' "$pane" | grep -q 'field commander\|esc to interrupt'; then
  118. log "bootstrap not confirmed in pane — retrying once"
  119. send_literal commander "$bootstrap"
  120. sleep 3
  121. fi
  122. log "commander TUI launched, bootstrap sent"
  123. fi
  124. # ---- 8. watchdog in the background, drive.log in the foreground ------------
  125. [[ -x "$WATCHDOG" ]] || die "watchdog missing: $WATCHDOG (image build broken?)"
  126. touch "$LOG"
  127. start_watchdog() {
  128. nohup bash "$WATCHDOG" >>"$LOG" 2>&1 &
  129. WATCHDOG_PID=$!
  130. log "watchdog started (pid $WATCHDOG_PID)"
  131. }
  132. END_AT=$(( $(date +%s) + TOTAL_MIN * 60 ))
  133. printf '%s\n' "$END_AT" > "$ROOT/.deadline" # single source of truth for the war clock
  134. MAX_WDG_RESPAWNS=3
  135. wdg_respawns=0
  136. start_watchdog
  137. tail -F -n0 "$LOG" &
  138. TAIL_PID=$!
  139. # Foreground supervisor: the watchdog is the guardian, this is its safety net.
  140. # The watchdog itself re-creates the tmux session at startup, so respawning it
  141. # also covers a vanished session. After the deadline a dead watchdog stays
  142. # dead — otherwise a respawn would hand itself a fresh full budget and never
  143. # let the container finish.
  144. while true; do
  145. sleep 15
  146. kill -0 "$WATCHDOG_PID" 2>/dev/null && continue
  147. if (( $(date +%s) >= END_AT )); then
  148. log "watchdog finished at deadline — campaign wrapped, container done"
  149. break
  150. fi
  151. if (( wdg_respawns >= MAX_WDG_RESPAWNS )); then
  152. log "FATAL: watchdog died $wdg_respawns times before the deadline — giving up"
  153. break
  154. fi
  155. wdg_respawns=$((wdg_respawns+1))
  156. log "WATCHDOG: process died unexpectedly — respawning ($wdg_respawns/$MAX_WDG_RESPAWNS)"
  157. start_watchdog
  158. done
  159. kill "$TAIL_PID" 2>/dev/null || true
docker/commander/watchdog.sh · Bundled script · 658 lines · 2 findings marked
  1. #!/usr/bin/env bash
  2. # watchdog.sh — the dumb guardian loop (commander architecture, SPEC unit 2).
  3. #
  4. # Failure modes handled (single sweep, sleep 15s, checks in SPEC order):
  5. # 1. commander pane dead (DRIVE_PANE_EXITED / window gone / no claude process)
  6. # -> relaunch claude in window "commander" (own cwd ROOT/commander-home)
  7. # with cwd-scoped --continue, wait for input box, inject bootstrap prompt
  8. # 2. bootstrap text stuck unsubmitted in the commander input box
  9. # -> re-Enter (3 tries), then restart the commander window
  10. # 3. trust/permission confirm dialogs in ANY pane -> auto-accept (Down+Enter)
  11. # 4. quota/auth errors in ANY pane -> rotate_key(); TUI workers and the
  12. # commander restart budget-free; -p workers only logged (the commander
  13. # notices via board). Rotation never burns restart budget.
  14. # 5. TUI worker DRIVE_PANE_EXITED -> auto-relaunch with cwd-scoped --continue,
  15. # max 3/pane (restarts map), budget-independent. -p worker exit = normal
  16. # completion: pane left for the commander (peek + worker.sh kill).
  17. # 6. global deadline (SAPPER_TOTAL_MINUTES from script start) -> C-c all
  18. # worker panes, finish_flags for EVERY code with a graph, arena close
  19. # every open instance, log summary, exit 0 (failsafe finish when the
  20. # commander is dead at the buzzer).
  21. # 7. worker TUI stall -> deliberately NOTHING (nudging is the commander's job).
  22. #
  23. # Explicitly NOT here (zero strategy): challenge selection, spawning new
  24. # challenges, strategy kills, mission text, nudging. Every relaunch reuses the
  25. # DRIVE_PANE_EXITED convention so death stays detectable. Relaunches never
  26. # re-send worker missions — the commander re-briefs amnesiac panes.
  27. set -o pipefail
  28. ROOT="${SAPPER_ROOT:-/work/runs}"
  29. LOG="$ROOT/drive.log"
  30. # Model tiers (SKILL 4.9): the commander runs SMART; each worker keeps the tier
  31. # it was spawned with (persisted by worker.sh in ROOT/pane-env/<pane>.env).
  32. CMD_MODEL="${SAPPER_MODEL:-glm-5.3[1m]}"
  33. FALLBACK_MODEL="${SAPPER_MODEL_FLASH:-glm-5.3-flash[1m]}"
  34. MODEL="$CMD_MODEL" # legacy var: current relaunch default, updated on rotation
  35. TOTAL_MIN="${SAPPER_TOTAL_MINUTES:-470}"
  36. MAX_WORKERS="${SAPPER_MAX_WORKERS:-8}"
  37. STALL_SECONDS="${SAPPER_STALL_SECONDS:-600}"
  38. SESSION=sapper-cmd
  39. CMD_HOME="$ROOT/commander-home"
  40. PANE_ENV_DIR="$ROOT/pane-env"
  41. SKILL_PATH=/opt/sapper/docker/commander/SKILL.md
  42. MAX_RESTARTS=3 # per TUI worker pane (SPEC unit 2 #5)
  43. mkdir -p "$ROOT" "$CMD_HOME" "$PANE_ENV_DIR"
  44. # stdout only: the entrypoint nohup-redirects us into drive.log (tee here
  45. # double-writes every line — observed in the sim log)
  46. log() { printf '[watch] %s\n' "$*"; }
  47. END_AT=$(( $(date +%s) + TOTAL_MIN * 60 ))
  48. NOW() { date +%s; }
  49. declare -A mode restarts enter_tries mission_restarts last_activity placeholder_since cooldown_log
  50. # ---- arena helper (verbatim from tmux-drive.sh) ------------------------------
  51. arena() {
  52. python3 - "$@" <<'PYEOF2'
  53. import importlib.util, json, sys
  54. spec = importlib.util.spec_from_file_location("arena", "/opt/sapper/lib/arena.py")
  55. m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
  56. c = m.ArenaClient.load(None)
  57. op = sys.argv[1]
  58. if op == "list":
  59. print(json.dumps(c.list_challenges()))
  60. elif op == "start":
  61. print(json.dumps(c.start_challenge(sys.argv[2])))
  62. elif op == "close":
  63. print(json.dumps(c.close_challenge(sys.argv[2])))
  64. PYEOF2
  65. }
  66. # ---- key-pool rotation ------------------------------------------------------
  67. # SAPPER_KEY_POOL: JSON [{base_url,token,model},...]. On quota/balance/auth
  68. # errors the watchdog rotates its own env; panes relaunched afterwards
  69. # inherit the new key, and `claude --continue` restores the prior session.
  70. # STORM GUARD (live incident 17:36-18:22): with every pool key dead the
  71. # rotation bounced every 16s and relaunched the whole fleet endlessly.
  72. # A 5-minute cooldown between rotations + a 6/hour cap turns that storm
  73. # into a slow drip; transient 429s ride claude's own retry instead.
  74. LAST_ROTATE=0
  75. ROTATIONS_THIS_HOUR=0
  76. HOUR_MARK=$(NOW)
  77. rotation_allowed() {
  78. local now; now=$(NOW)
  79. if (( now - HOUR_MARK > 3600 )); then HOUR_MARK=$now; ROTATIONS_THIS_HOUR=0; fi
  80. if (( ROTATIONS_THIS_HOUR >= 6 )); then
  81. cooldown_log pool-exhausted "rotation cap (6/h) hit — pool likely fully dead; pausing rotations this hour"
  82. return 1
  83. fi
  84. if (( now - LAST_ROTATE < 300 )); then
  85. cooldown_log rot-cooldown "rotation cooldown (5min) — transient errors should ride claude's own retry"
  86. return 1
  87. fi
  88. return 0
  89. }
  90. rotate_key() {
  91. python3 - <<'PYROT'
  92. import json, os
  93. raw = os.environ.get("SAPPER_KEY_POOL", "").strip()
  94. try:
  95. pool = [p for p in json.loads(raw) if isinstance(p, dict) and p.get("token")]
  96. except Exception:
  97. pool = []
  98. if not pool:
  99. raise SystemExit(1)
  100. def fam(e):
  101. u = (e.get("base_url") or "") + " " + (e.get("model") or "")
  102. if "bigmodel" in u or "glm" in u: return "glm"
  103. if "deepseek" in u: return "ds"
  104. if "moonshot" in u or "kimi" in u: return "kimi"
  105. return "other"
  106. cur = os.environ.get("ANTHROPIC_AUTH_TOKEN", "")
  107. cur_fam = next((fam(e) for e in pool if e["token"] == cur), None)
  108. if cur_fam is None: # main key not in pool: classify by its base_url
  109. b = os.environ.get("ANTHROPIC_BASE_URL", "")
  110. cur_fam = "glm" if "bigmodel" in b else ("ds" if "deepseek" in b else "other")
  111. # tried-memory (1h window): a family whose every key already failed this hour
  112. # is skipped, so two dead glm keys bounce K1<->K2 forever instead of crossing
  113. # to deepseek. All tried -> quiet stop; the window clears in an hour.
  114. import time
  115. root = os.environ.get("SAPPER_ROOT", "/work/runs")
  116. tried_path = root + "/.rotated-tried.txt"
  117. now = time.time()
  118. tried = []
  119. try:
  120. for ln in open(tried_path):
  121. ts, tk = ln.strip().split(" ", 1)
  122. if now - float(ts) < 3600:
  123. tried.append(tk)
  124. except Exception:
  125. pass
  126. same = [e for e in pool if fam(e) == cur_fam]
  127. # same-family FIRST, untried first: a glm pane must never be flung onto
  128. # deepseek while an untried glm key still lives (the mismatch crash-loop)
  129. cands = [e for e in same if e["token"] != cur and e["token"] not in tried]
  130. if not cands:
  131. cands = [e for e in same if e["token"] != cur] # family exhausted: retry oldest? no—cross below
  132. cands = []
  133. if not cands:
  134. cands = [e for e in pool if e["token"] != cur and e["token"] not in tried]
  135. if not cands:
  136. raise SystemExit(1) # everything tried this hour: quiet stop
  137. nxt = cands[0]
  138. model = nxt.get("model") or os.environ.get("SAPPER_MODEL", "")
  139. base = nxt.get("base_url") or os.environ.get("ANTHROPIC_BASE_URL", "")
  140. root = os.environ.get("SAPPER_ROOT", "/work/runs")
  141. with open(root + "/rotated.env", "w") as fh: # global (commander follows this)
  142. fh.write(f'export ANTHROPIC_BASE_URL="{base}"\n'
  143. f'export ANTHROPIC_AUTH_TOKEN="{nxt["token"]}"\n'
  144. f'export SAPPER_MODEL="{model}"\n')
  145. with open(f"{root}/rotated-{fam(nxt)}.env", "w") as fh: # per-family current creds
  146. fh.write(f'ANTHROPIC_BASE_URL={base}\nANTHROPIC_AUTH_TOKEN={nxt["token"]}\nMODEL={model}\n')
  147. if cur and any(e["token"] == cur for e in pool):
  148. with open(tried_path, "a") as fh:
  149. fh.write(f"{now} {cur}\n")
  150. raise SystemExit(0)
  151. PYROT
  152. local rc=$?
  153. if (( rc == 0 )) && [[ -f "$ROOT/rotated.env" ]]; then
  154. LAST_ROTATE=$(NOW); ROTATIONS_THIS_HOUR=$((ROTATIONS_THIS_HOUR + 1))
  155. local old_url="$ANTHROPIC_BASE_URL"
  156. # shellcheck disable=SC1090
  157. . "$ROOT/rotated.env"; export ANTHROPIC_BASE_URL ANTHROPIC_AUTH_TOKEN SAPPER_MODEL
  158. MODEL="${SAPPER_MODEL:-$MODEL}" # watchdog addition: relaunches must pick up the rotated model
  159. if [[ -n "$old_url" && "$old_url" != "$ANTHROPIC_BASE_URL" ]]; then
  160. # cross-family rotation (e.g. glm exhausted -> deepseek). Do NOT rewrite
  161. # every pane-env: a glm pane flung onto the deepseek endpoint with a glm
  162. # model id crashes instantly and re-triggers rotation (the 19:04 storm).
  163. # glm panes keep their identity and simply ride the rotation cooldown;
  164. # the commander can re-dispatch them on --gw ds if it chooses.
  165. CMD_MODEL="${SAPPER_MODEL:-$CMD_MODEL}" # commander relaunches must answer on the new provider too
  166. log "key rotated cross-family -> $ANTHROPIC_BASE_URL (commander follows; glm panes keep glm creds)"
  167. else
  168. log "key rotated -> $ANTHROPIC_BASE_URL (model $SAPPER_MODEL)"
  169. fi
  170. return 0
  171. fi
  172. return 1
  173. }
  174. QUOTA_RE='insufficient|balance|quota|exceeded|billing|payment|401|[Uu]nauthorized|invalid.{0,12}[Aa][Pp][Ii].[Kk]ey|欠费|余额'
  175. # transient rate limits look quota-ish but mean the key is FINE (live root
  176. # cause of the 17:36 storm: per-minute limits were misread as a dead key)
  177. RATELIMIT_RE='每分钟|分钟级|rate.?limit|too many requests|429|1302|1311|1305'
  178. # submit every recorded-unsubmitted flag fact in a challenge's graph (verbatim)
  179. finish_flags() { # finish_flags <code>
  180. python3 - "$1" <<'PYFIN'
  181. import glob, json, os, sys, importlib.util
  182. code = sys.argv[1]
  183. root = os.environ.get("SAPPER_ROOT", "/work/runs")
  184. spec = importlib.util.spec_from_file_location("arena", "/opt/sapper/lib/arena.py")
  185. m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
  186. try:
  187. client = m.ArenaClient.load(None)
  188. except Exception as exc:
  189. print(f"[finish] {code}: no arena ({exc})"); sys.exit(0)
  190. for path in sorted(glob.glob(f"{root}/{code}/.sapper/facts/*.json")):
  191. try:
  192. fact = json.load(open(path))
  193. except Exception:
  194. continue
  195. if fact.get("kind") != "flag":
  196. continue
  197. data = fact.get("data") or {}
  198. if data.get("status") not in (None, "recorded") or not data.get("value"):
  199. continue
  200. try:
  201. r = client.submit_flag(code, data["value"])
  202. fact["data"]["status"] = "submitted" if r.get("correct") else "rejected"
  203. json.dump(fact, open(path, "w"), ensure_ascii=False, indent=2)
  204. print(f"[finish] {code}: submit {data['value'][:24]}… correct={r.get('correct')}")
  205. except Exception as exc:
  206. msg = str(exc)
  207. if "already" in msg.lower() or "409" in msg:
  208. # duplicate submit = the flag WAS accepted earlier: absorb it
  209. fact["data"]["status"] = "submitted"
  210. json.dump(fact, open(path, "w"), ensure_ascii=False, indent=2)
  211. print(f"[finish] {code}: already submitted — marked ({data['value'][:20]}…)")
  212. else:
  213. print(f"[finish] {code}: submit failed ({exc})")
  214. PYFIN
  215. }
  216. # ---- trust preset (PYTRUST + settings.json, from tmux-drive.sh launch()) ----
  217. preset_trust() { # preset_trust <dir> — idempotent; runs BEFORE a pane (re)spawn
  218. local dir="$1"
  219. mkdir -p "$dir/.claude"
  220. printf '%s\n' '{"permissions":{"defaultMode":"bypassPermissions"},"autoCompactEnabled":true}' > "$dir/.claude/settings.json"
  221. python3 - "$dir" <<'PYTRUST'
  222. import json, os, sys
  223. path = os.path.expanduser("~/.claude.json")
  224. try: d = json.load(open(path))
  225. except Exception: d = {}
  226. d.setdefault("projects", {})[sys.argv[1]] = {
  227. "hasTrustDialogAccepted": True, "bypassPermissionsModeAccepted": True}
  228. json.dump(d, open(path, "w"))
  229. PYTRUST
  230. }
  231. # ---- tmux primitives (verbatim from tmux-drive.sh) ---------------------------
  232. pane_text() { tmux capture-pane -pt "$SESSION:$1" 2>/dev/null; }
  233. send_literal() { # send_literal <window> <text> [enter]
  234. tmux send-keys -t "$SESSION:$1" -l -- "$2"
  235. sleep 0.5
  236. [[ "${3:-enter}" == "enter" ]] && tmux send-keys -t "$SESSION:$1" Enter
  237. }
  238. accept_dialogs() { # accept_dialogs <window> <pane> -> 0 if a dialog was dismissed
  239. local win="$1" pane="$2"
  240. if printf '%s' "$pane" | grep -qE 'Enter to confirm|Yes, I accept|I accept'; then
  241. # Aim by the ❯ focus marker: security dialogs default-focus "No, exit",
  242. # so a bare Enter CHOOSES NO. Only move+confirm when focus is off Yes.
  243. if printf '%s' "$pane" | grep -qE '❯[^\n]*No, exit'; then
  244. tmux send-keys -t "$SESSION:$win" Down; sleep 0.4
  245. fi
  246. tmux send-keys -t "$SESSION:$win" Enter; sleep 1.5
  247. return 0
  248. fi
  249. return 1
  250. }
  251. # ---- relaunch machinery ------------------------------------------------------
  252. sync_session_env() { # tmux panes inherit the SESSION environment, NOT this
  253. # process's env (only update-environment vars are copied
  254. # from clients). After rotate_key() mutates our env, the
  255. # rotated key MUST be pushed into the session or every
  256. # relaunched pane boots claude on the stale dead key and
  257. # re-triggers the quota error forever.
  258. local k v
  259. for k in PATH ANTHROPIC_BASE_URL ANTHROPIC_AUTH_TOKEN SAPPER_MODEL \
  260. SAPPER_MODEL_FLASH BENCHMARK_BASE_URL BENCHMARK_TOKEN; do
  261. v=${!k:-}
  262. [[ -n "$v" ]] && tmux set-environment -t "$SESSION" "$k" "$v" >/dev/null 2>&1 || true
  263. done
  264. }
  265. DS_MODEL="${SAPPER_MODEL_DS:-deepseek-flash}"
  266. DS_BASE_URL="${DEEPSEEK_BASE_URL:-https://api.deepseek.com/anthropic}"
  267. DS_KEY="${DEEPSEEK_API_KEY:-}"
  268. pane_env_get() { # pane_env_get <win> <KEY> — RAW value from pane-env/<win>.env
  269. [[ -f "$PANE_ENV_DIR/$1.env" ]] && sed -n "s/^$2=//p" "$PANE_ENV_DIR/$1.env" 2>/dev/null | head -1
  270. }
  271. pane_model() { # pane_model <win> — the tier this worker was spawned with
  272. local m; m=$(pane_env_get "$1" MODEL)
  273. printf '%s' "${m:-$FALLBACK_MODEL}"
  274. }
  275. claude_resume_flag() { # sets RESUME_FLAG="--continue" iff <cwd> has a prior session
  276. RESUME_FLAG=""
  277. # glob is only a has-this-cwd-run heuristic; claude resolves the session by
  278. # real cwd, so a loose match can never resume the WRONG session
  279. ls "$HOME/.claude/projects/"*"$(basename "$1")"*/ >/dev/null 2>&1 && RESUME_FLAG="--continue"
  280. return 0
  281. }
  282. wait_input_box() { # wait_input_box <win> — 0 when '❯' shows (auto-accepting dialogs)
  283. local win="$1" waited=0 pane=""
  284. while (( waited < 75 )); do
  285. pane=$(pane_text "$win")
  286. accept_dialogs "$win" "$pane" && { waited=$((waited+3)); continue; }
  287. printf '%s' "$pane" | grep -q '❯' && return 0
  288. sleep 1; waited=$((waited+1))
  289. done
  290. log "$win: input box never appeared"
  291. return 1
  292. }
  293. relaunch_pane() { # relaunch_pane <win> <cwd> <model> — claude TUI, same window, no mission text
  294. local win="$1" cwd="$2" model="$3" cmd gw engine gw_env
  295. preset_trust "$cwd"
  296. claude_resume_flag "$cwd"
  297. sync_session_env
  298. # worker panes carry their gateway/engine in pane-env; the commander is always claude+glm-family.
  299. # Inject THIS pane's FAMILY credentials: same-family rotation keeps a glm pane
  300. # on glm keys even when the global env (commander) has already crossed to
  301. # another provider — the mismatch crash-loop fix.
  302. engine=$(pane_env_get "$win" ENGINE)
  303. # NO family pinning anywhere (user doctrine: nothing is pinned). Every pane
  304. # follows the ROTATED GLOBAL credentials via the tmux session env
  305. # (sync_session_env pushes them after each rotation). Model follows too:
  306. # commander uses CMD_MODEL, workers use SAPPER_MODEL — both updated on
  307. # rotation, so a cross-family slide migrates key AND model together.
  308. # (The old per-family pin once handed the commander a dead glm key while its
  309. # model had already crossed to deepseek — a 15-min bootstrap boot-lock.)
  310. gw_env=""
  311. if [[ "$engine" == codex ]]; then
  312. local provider=deepseek; [[ "$DS_BASE_URL" == https://api.deepseek.com* ]] && provider=deepseek-local
  313. cmd="cd '$cwd' && DEEPSEEK_API_KEY='$DS_KEY' codex --skip-git-repo-check -c model_provider=$provider -c model='$model' --dangerously-bypass-approvals-and-sandbox 2>&1; echo DRIVE_PANE_EXITED; exec sleep infinity"
  314. else
  315. cmd="cd '$cwd' && $gw_env claude $RESUME_FLAG --model '$model' --effort max --dangerously-skip-permissions 2>&1; echo DRIVE_PANE_EXITED; exec sleep infinity"
  316. fi
  317. # archived mission (TUI workers): re-brief after the window comes back —
  318. # a fresh post-restart session would otherwise sit at an empty prompt
  319. REBRIEF_TEXT=""
  320. if [[ "$win" != "commander" ]] && command -v base64 >/dev/null 2>&1; then
  321. local rb; rb=$(pane_env_get "$win" MISSION_B64)
  322. [[ -n "$rb" ]] && REBRIEF_TEXT=$(printf '%s' "$rb" | base64 -d 2>/dev/null || true)
  323. fi
  324. if tmux list-windows -t "$SESSION" -F '#W' 2>/dev/null | grep -qx -- "$win"; then
  325. # window still exists: respawn in place (kill-window on a session's last
  326. # window would destroy the session — and the guardian with it)
  327. tmux respawn-pane -k -t "$SESSION:$win" "$cmd" 2>>"$LOG" || { log "$win: respawn failed"; return 1; }
  328. else
  329. tmux new-window -d -t "$SESSION" -n "$win" "$cmd" 2>>"$LOG" || { log "$win: new-window failed"; return 1; }
  330. fi
  331. last_activity[$win]=$(NOW)
  332. wait_input_box "$win"
  333. # re-brief: the pane's archived mission (fresh session post-restart would
  334. # otherwise sit at an empty prompt forever, never calling the LLM)
  335. if [[ -n "${REBRIEF_TEXT:-}" ]]; then
  336. tmux send-keys -t "$SESSION:$win" C-u 2>/dev/null; sleep 0.3
  337. send_literal "$win" "$REBRIEF_TEXT"
  338. log "$win: archived mission re-sent after relaunch"
  339. REBRIEF_TEXT=""
  340. fi
  341. }
  342. relaunch_commander() { # relaunch_commander <reason> — SPEC unit 2 #1
  343. local reason="$1" verb
  344. log "commander: relaunching ($reason, cwd $CMD_HOME)"
  345. mkdir -p "$CMD_HOME"
  346. if relaunch_pane commander "$CMD_HOME" "$CMD_MODEL"; then
  347. verb="start commanding"
  348. [[ -f "$ROOT/commander-state.md" ]] && verb="resume command"
  349. # clear any stale text from a previous unsubmitted bootstrap first (else
  350. # send_literal concatenates onto it — seen live as doubled input text)
  351. tmux send-keys -t "$SESSION:commander" C-u 2>/dev/null
  352. sleep 0.3
  353. send_literal commander \
  354. "Read $SKILL_PATH and $verb. First read $ROOT/commander-state.md if it exists (your handover from a previous incarnation)."
  355. last_activity[commander]=$(NOW)
  356. log "commander: bootstrap prompt sent ($verb)"
  357. fi
  358. enter_tries[commander]=0
  359. mode[commander]=commander
  360. placeholder_since[commander]=0
  361. }
  362. relaunch_worker() { # relaunch_worker <win> <reason> — cwd-scoped --continue, no mission
  363. local win="$1" reason="$2" cwd
  364. # the pane's `cd` survives death (exec sleep infinity keeps the shell cwd)
  365. cwd=$(tmux display-message -p -t "$SESSION:$win" '#{pane_current_path}' 2>/dev/null)
  366. if [[ -z "$cwd" || ! -d "$cwd" ]]; then
  367. log "$win: cannot determine cwd — leaving dead pane for the commander"
  368. return 1
  369. fi
  370. local m="${SAPPER_MODEL:-$(pane_model "$win")}"
  371. log "$win: relaunching TUI with --continue ($reason, cwd $cwd, model $m)"
  372. relaunch_pane "$win" "$cwd" "$m" || return 1
  373. mode[$win]=tui
  374. return 0
  375. }
  376. adopt_existing() { # adopt windows we did not launch ourselves (worker.sh / entrypoint)
  377. local win cmd
  378. for win in $(tmux list-windows -t "$SESSION" -F '#W' 2>/dev/null || true); do
  379. [[ -n "${last_activity[$win]:-}" ]] && continue
  380. cmd=$(tmux display-message -p -t "$SESSION:$win" '#{pane_start_command}' 2>/dev/null)
  381. if [[ "$win" == "commander" ]]; then mode[$win]=commander
  382. elif printf '%s' "$cmd" | grep -q 'sapper auto'; then mode[$win]=p
  383. else mode[$win]=tui; fi
  384. last_activity[$win]=$(NOW); restarts[$win]=0
  385. log "$win: adopted existing window (mode=${mode[$win]})"
  386. done
  387. }
  388. ensure_session() {
  389. tmux has-session -t "$SESSION" 2>/dev/null && return 0
  390. log "tmux session $SESSION missing — recreating"
  391. tmux new-session -d -s "$SESSION" -n commander 2>>"$LOG" || { log "FATAL: cannot recreate tmux session"; return 1; }
  392. return 0
  393. }
  394. cooldown_log() { # cooldown_log <key> <msg> — drive.log is also the operator console: no spam
  395. local key="$1" msg="$2" now; now=$(NOW)
  396. if (( now - ${cooldown_log[$key]:-0} > 300 )); then
  397. cooldown_log[$key]=$now
  398. log "$msg"
  399. fi
  400. }
  401. finish_all() { # SPEC unit 2 #6 — global deadline failsafe
  402. local win g real code n_codes=0 n_closed=0
  403. log "global deadline reached — failsafe finish"
  404. for win in $(tmux list-windows -t "$SESSION" -F '#W' 2>/dev/null || true); do
  405. [[ "$win" == "commander" ]] && continue
  406. log "$win: C-c (deadline)"
  407. tmux send-keys -t "$SESSION:$win" C-c 2>/dev/null
  408. done
  409. sleep 3
  410. # every code with a graph; follow -wN .sapper symlinks back to the primary
  411. # code so shared graphs are finished exactly once
  412. local -A seen=()
  413. for g in "$ROOT"/*/.sapper; do
  414. [[ -e "$g" ]] || continue
  415. real=$(realpath "$g" 2>/dev/null) || continue
  416. code=$(basename "$(dirname "$real")")
  417. [[ -n "${seen[$code]:-}" ]] && continue
  418. seen[$code]=1; n_codes=$((n_codes+1))
  419. log "$code: deadline finish_flags"
  420. finish_flags "$code" 2>&1 | tee -a "$LOG"
  421. done
  422. # close every open platform instance (409/other errors must never crash us)
  423. for code in $(arena list 2>>"$LOG" | python3 -c '
  424. import json,sys
  425. try: items=json.load(sys.stdin)
  426. except Exception: sys.exit(0)
  427. for c in items:
  428. if isinstance(c,dict) and not c.get("is_completed"): print(c["unique_code"])' 2>/dev/null); do
  429. if arena close "$code" >/dev/null 2>&1; then
  430. log "$code: instance closed"; n_closed=$((n_closed+1))
  431. else
  432. log "$code: instance close failed (ignored)"
  433. fi
  434. done
  435. log "watchdog complete: graphs_finished=$n_codes instances_closed=$n_closed model=$MODEL"
  436. exit 0
  437. }
  438. # ---- startup -----------------------------------------------------------------
  439. keys=$(printf '%s' "${SAPPER_KEY_POOL:-}" | python3 -c '
  440. import json,sys
  441. try: print(len([p for p in json.load(sys.stdin) if isinstance(p,dict) and p.get("token")]))
  442. except Exception: print(0)' 2>/dev/null || echo 0)
  443. log "watchdog start: root=$ROOT session=$SESSION cmd_home=$CMD_HOME"
  444. printf '%s\n' "$END_AT" > "$ROOT/.deadline" # board reads this: budget_left heartbeat
  445. log "env: model=$MODEL total=${TOTAL_MIN}m max_workers=$MAX_WORKERS stall=${STALL_SECONDS}s(informational) keys=$keys"
  446. log "env: benchmark=${BENCHMARK_BASE_URL:-unset} gateway=${ANTHROPIC_BASE_URL:-unset} deadline=$(date -d "@$END_AT" '+%F %T' 2>/dev/null || echo "in ${TOTAL_MIN}m")"
  447. # family-initial credentials: what relaunch_pane falls back to before any rotation
  448. GLM_BASE_URL_INITIAL="${ANTHROPIC_BASE_URL:-}"
  449. GLM_TOKEN_INITIAL="${ANTHROPIC_AUTH_TOKEN:-}"
  450. fresh_session=0
  451. if ! tmux has-session -t "$SESSION" 2>/dev/null; then
  452. # entrypoint normally owns first boot; reaching this means it never ran
  453. # (manual start) or tmux died — the watchdog must be self-sufficient
  454. tmux new-session -d -s "$SESSION" -n commander 2>>"$LOG" || { log "FATAL: cannot create tmux session $SESSION"; exit 1; }
  455. fresh_session=1
  456. log "created tmux session $SESSION (window 0 = commander)"
  457. fi
  458. adopt_existing
  459. if (( fresh_session )); then
  460. relaunch_commander "fresh session, no commander running"
  461. elif ! tmux list-windows -t "$SESSION" -F '#W' 2>/dev/null | grep -qx commander; then
  462. relaunch_commander "commander window absent at startup"
  463. fi
  464. # ---- main loop ----------------------------------------------------------------
  465. while (( $(NOW) < END_AT )); do
  466. ensure_session || { sleep 15; continue; }
  467. adopt_existing
  468. wins=$(tmux list-windows -t "$SESSION" -F '#W' 2>/dev/null || true)
  469. # -- 1) commander pane dead -> relaunch + bootstrap --------------------------
  470. cmd_handled=0
  471. if ! printf '%s\n' "$wins" | grep -qx commander; then
  472. relaunch_commander "commander window gone"
  473. cmd_handled=1
  474. else
  475. cpane=$(pane_text commander)
  476. if printf '%s' "$cpane" | grep -q 'DRIVE_PANE_EXITED'; then
  477. relaunch_commander "commander pane exited"
  478. cmd_handled=1
  479. fi
  480. fi
  481. # -- 1b) DELETED: the pane_current_command whitelist (claude|node) was
  482. # unreliable across claude-code builds — it false-positived on the
  483. # LIVE commander TUI every ~60s and boot-looped it on the hosted run
  484. # (input box existed => TUI alive, yet "no claude process" fired).
  485. # Death is now judged ONLY by the iron signals in check 1:
  486. # DRIVE_PANE_EXITED on screen, or the window gone.
  487. # -- 2) unsubmitted bootstrap text in the input box -> re-Enter x3 -> restart
  488. # (matches both the watchdog's and the entrypoint's bootstrap text)
  489. if (( ! cmd_handled )); then
  490. if printf '%s' "$cpane" | grep -qE "❯.*docker/commander/SKILL\.md"; then
  491. enter_tries[commander]=$(( ${enter_tries[commander]:-0} + 1 ))
  492. case $(( ${enter_tries[commander]} )) in
  493. 1|2)
  494. log "commander: input unsubmitted — re-Enter (${enter_tries[commander]}/3)"
  495. tmux send-keys -t "$SESSION:commander" Enter
  496. sleep 2
  497. ;;
  498. 3)
  499. # stuck twice: clear the box and retype the whole bootstrap (the
  500. # text may be mangled mid-line; fresh keystrokes beat more Enters)
  501. log "commander: input unsubmitted — clearing and resending bootstrap (3/3)"
  502. tmux send-keys -t "$SESSION:commander" C-u
  503. sleep 0.5
  504. tmux send-keys -t "$SESSION:commander" C-u
  505. sleep 0.5
  506. tmux send-keys -t "$SESSION:commander" Enter
  507. sleep 1
  508. relaunch_commander "bootstrap never submits cleanly"
  509. ;;
  510. esac
  511. last_activity[commander]=$(NOW)
  512. cmd_handled=1
  513. else
  514. enter_tries[commander]=0
  515. fi
  516. fi
  517. # -- 2b) commander hang failsafe: pane content unchanged for 30 min (and
  518. # nothing waiting in the input box) -> the TUI is wedged, not thinking.
  519. # A dead-silent commander for the rest of the slot is the worst
  520. # outcome; a relaunch costs one --continue.
  521. if (( ! cmd_handled )); then
  522. csha=$(printf '%s' "$cpane" | sha256sum | cut -c1-12)
  523. if [[ "${cmd_pane_sha:-}" == "$csha" && -n "$csha" ]]; then
  524. if (( $(NOW) - ${cmd_pane_since:-$(NOW)} > 1800 )) \
  525. && ! printf '%s' "$cpane" | grep -qE '❯.+[^ ]'; then
  526. log "commander: pane frozen 30min with empty input — relaunching (hang failsafe)"
  527. relaunch_commander "hang failsafe"
  528. cmd_handled=1
  529. fi
  530. else
  531. cmd_pane_sha="$csha"; cmd_pane_since=$(NOW)
  532. fi
  533. fi
  534. # -- 3+4) commander: dialogs first, then quota -------------------------------
  535. if (( ! cmd_handled )); then
  536. if accept_dialogs commander "$cpane"; then
  537. last_activity[commander]=$(NOW); cmd_handled=1
  538. elif printf '%s' "$cpane" | grep -qE "$QUOTA_RE"; then
  539. if printf '%s' "$cpane" | grep -qE "$RATELIMIT_RE"; then
  540. cooldown_log commander-rl "transient rate limit (not quota) — no rotation; claude retries on its own"
  541. elif rotation_allowed && rotate_key; then
  542. cooldown_log commander "quota/auth error in commander — key rotated, relaunching"
  543. relaunch_commander "quota/auth error"
  544. else
  545. cooldown_log commander "quota/auth error in commander and no spare key — waiting"
  546. fi
  547. last_activity[commander]=$(NOW); cmd_handled=1
  548. fi
  549. fi
  550. # -- 3,4,5) worker panes ------------------------------------------------------
  551. for win in $wins; do
  552. [[ "$win" == "commander" ]] && continue
  553. pane=$(pane_text "$win")
  554. # 3) any confirm dialog -> accept
  555. if accept_dialogs "$win" "$pane"; then
  556. last_activity[$win]=$(NOW); continue
  557. fi
  558. # 4) quota/auth -> rotate; TUI restarts budget-free, -p only logged
  559. if printf '%s' "$pane" | grep -qE "$QUOTA_RE"; then
  560. if printf '%s' "$pane" | grep -qE "$RATELIMIT_RE"; then
  561. cooldown_log "$win-rl" "$win: transient rate limit (not quota) — no rotation; claude retries on its own"
  562. last_activity[$win]=$(NOW); continue
  563. fi
  564. if rotation_allowed && rotate_key; then
  565. if [[ "${mode[$win]:-tui}" == "p" ]]; then
  566. cooldown_log "$win" "quota/auth error in -p pane — key rotated; leaving pane (commander will notice via board)"
  567. else
  568. cooldown_log "$win" "quota/auth error — key rotated, relaunching TUI with --continue (restart budget untouched)"
  569. relaunch_worker "$win" "quota/auth error" || true
  570. fi
  571. else
  572. cooldown_log "$win" "quota/auth error and no spare key — leaving pane as-is"
  573. fi
  574. last_activity[$win]=$(NOW); continue
  575. fi
  576. # 4b) TUI worker mission stuck unsubmitted in the input box -> re-Enter,
  577. # then restart the window (regression from tmux-drive.sh #3: the
  578. # hosted gateway is slow and paste-loss strands the mission text,
  579. # leaving a live-looking pane that NEVER calls the LLM)
  580. if [[ "${mode[$win]:-tui}" == "tui" ]] \
  581. && printf '%s' "$pane" | grep -qE '❯.*autonomous pentest operator'; then
  582. enter_tries[$win]=$(( ${enter_tries[$win]:-0} + 1 ))
  583. if (( ${enter_tries[$win]} > 3 )); then
  584. mission_restarts[$win]=$(( ${mission_restarts[$win]:-0} + 1 ))
  585. if (( ${mission_restarts[$win]} > 3 )); then
  586. # restart loops help nothing (live: 25min of re-Enter/restart cycling
  587. # on a wedged TUI) — stop touching it, leave the diagnosis to the
  588. # commander (peek + kill/park). Cooldown-log to avoid spam.
  589. cooldown_log "$win-ms" "$win: mission wedged past 3 restarts — leaving pane for the commander"
  590. last_activity[$win]=$(NOW)
  591. continue
  592. fi
  593. log "$win: mission never submits — restarting window (${mission_restarts[$win]}/3)"
  594. enter_tries[$win]=0
  595. relaunch_worker "$win" "mission never submits" || true
  596. else
  597. log "$win: mission unsubmitted — re-Enter (${enter_tries[$win]}/3)"
  598. tmux send-keys -t "$SESSION:$win" Enter
  599. sleep 2
  600. fi
  601. last_activity[$win]=$(NOW)
  602. continue
  603. fi
  604. # 5) TUI worker exited -> relaunch with --continue, max 3/pane; -p = done
  605. if printf '%s' "$pane" | grep -q 'DRIVE_PANE_EXITED'; then
  606. if [[ "${mode[$win]:-tui}" == "p" ]]; then
  607. cooldown_log "$win" "$win: -p run exited — leaving pane for the commander (peek + worker.sh kill)"
  608. continue
  609. fi
  610. restarts[$win]=$(( ${restarts[$win]:-0} + 1 ))
  611. if (( ${restarts[$win]} > MAX_RESTARTS )); then
  612. cooldown_log "$win" "$win: TUI exited with no restarts left ($MAX_RESTARTS used) — leaving dead pane for the commander"
  613. continue
  614. fi
  615. relaunch_worker "$win" "TUI exited (${restarts[$win]}/$MAX_RESTARTS)" || true
  616. continue
  617. fi
  618. # 7) stall (no pane change > SAPPER_STALL_SECONDS): deliberately NO nudge,
  619. # NO restart — keeping processes alive is the watchdog's whole job
  620. done
  621. sleep 15
  622. done
  623. finish_all
  624. exit 0
docker/commander/worker.sh · Bundled script · 776 lines · 3 findings marked
  1. #!/usr/bin/env bash
  2. # worker.sh v1 — the commander's hands: one-shot subcommands, no main loop.
  3. #
  4. # Failure modes handled:
  5. # - platform "max active" cap (HTTP 409) -> detected on arena start, friendly
  6. # PLATFORM CAP message + exit 3; every arena call is guarded so a 409 never
  7. # crashes the caller (inherited trap: "arena HTTP 409 must never crash")
  8. # - worker pane cap -> live window count vs
  9. # SAPPER_MAX_WORKERS enforced before any platform call, exit 4 WORKER CAP
  10. # - trust/bypass/permission dialogs -> per-project hasTrustDialogAccepted
  11. # preset + Down+Enter auto-accept while waiting for the input box
  12. # - bracketed paste breaks ink TUIs -> send-keys -l ONLY, never
  13. # load-buffer/paste-buffer
  14. # - mission stuck unsubmitted in input box-> confirm text in pane, retry once
  15. # - pane death detection -> every worker window ends with
  16. # "echo DRIVE_PANE_EXITED; exec sleep infinity" (list/board/watchdog read it)
  17. # - same-cwd --continue cross-talk -> one dir per agent; -w2+ mounts the
  18. # shared graph via a .sapper symlink instead of a second graph
  19. # - instance active but panes all dead -> container addrs persisted to
  20. # ROOT/<code>/.addrs and reused when arena start refuses/duplicates
  21. # - rotated gateway keys (watchdog) -> spawn syncs the tmux SESSION env so
  22. # new panes inherit the CURRENT key, not the server-startup key (tmux only
  23. # copies update-environment vars from the client — ANTHROPIC_* is not in it)
  24. # - repeated calls -> idempotent: kill-window before
  25. # create, guarded sapper init, next free -wN index derived from live panes
  26. set -o pipefail
  27. ROOT="${SAPPER_ROOT:-/work/runs}"
  28. LOG="$ROOT/drive.log"
  29. # Model tiers & gateways (SKILL 4.9, final doctrine): ALL-IN FLAGSHIP.
  30. # glm gw + SMART (glm-5.3[1m]) is the default brain for EVERY worker — soldier,
  31. # scout, chore, general alike. flash is retired from this deployment.
  32. # ds gw (deepseek-flash) is the RESERVE army: takes over when glm quota/
  33. # rate limits bite (the watchdog rotates; the commander may also spawn
  34. # --gw ds deliberately under quota pressure).
  35. # mixed doctrine: soldiers default to the ds tier (cheap mass), generals and
  36. # the commander run SAPPER_MODEL (the strong brain) — the general-escalation
  37. # below fires when model==MODEL, which holds for this default.
  38. # NOTE: inline SAPPER_MODEL_DS (DS_MODEL is defined LATER in this header —
  39. # referencing it here silently fell back to glm and killed the ds army).
  40. MODEL="${SAPPER_MODEL_DS:-${SAPPER_MODEL:-glm-5.3[1m]}}"
  41. DS_MODEL="${SAPPER_MODEL_DS:-deepseek-flash}"
  42. DS_BASE_URL="${DEEPSEEK_BASE_URL:-https://api.deepseek.com/anthropic}"
  43. DS_KEY="${DEEPSEEK_API_KEY:-}"
  44. SESSION=sapper-cmd
  45. MAX_WORKERS="${SAPPER_MAX_WORKERS:-8}"
  46. DEF_MINUTES=10
  47. SAPPER="${SAPPER_BIN:-sapper}"
  48. ACT_FILE="$ROOT/.worker-activity.tsv"
  49. PANE_ENV_DIR="$ROOT/pane-env"
  50. NOW() { date +%s; }
  51. mkdir -p "$ROOT" "$PANE_ENV_DIR"
  52. log() { printf '[worker] %s\n' "$*" | tee -a "$LOG" >&2; }
  53. die() { local rc=$1; shift; printf '[worker] ERROR: %s\n' "$*" >&2; exit "$rc"; }
  54. is_code() { [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9_.:-]*$ ]]; }
  55. window_exists() { tmux list-windows -t "$SESSION" -F '#W' 2>/dev/null | grep -qx -- "$1"; }
  56. # ---- arena helper (verbatim from tmux-drive.sh; module path env-overridable
  57. # for host-side testing — in-image default is identical) ------------------
  58. arena() {
  59. python3 - "$@" <<'PYEOF2'
  60. import importlib.util, json, os, sys
  61. spec = importlib.util.spec_from_file_location(
  62. "arena", os.environ.get("SAPPER_ARENA_LIB", "/opt/sapper/lib/arena.py"))
  63. m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
  64. c = m.ArenaClient.load(None)
  65. op = sys.argv[1]
  66. if op == "list":
  67. print(json.dumps(c.list_challenges()))
  68. elif op == "start":
  69. print(json.dumps(c.start_challenge(sys.argv[2])))
  70. elif op == "close":
  71. print(json.dumps(c.close_challenge(sys.argv[2])))
  72. PYEOF2
  73. }
  74. # submit every recorded-unsubmitted flag fact in a challenge's graph
  75. # (verbatim from tmux-drive.sh; module path env-overridable as above)
  76. finish_flags() { # finish_flags <code>
  77. python3 - "$1" <<'PYFIN'
  78. import glob, json, os, sys, importlib.util
  79. code = sys.argv[1]
  80. root = os.environ.get("SAPPER_ROOT", "/work/runs")
  81. spec = importlib.util.spec_from_file_location(
  82. "arena", os.environ.get("SAPPER_ARENA_LIB", "/opt/sapper/lib/arena.py"))
  83. m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
  84. try:
  85. client = m.ArenaClient.load(None)
  86. except Exception as exc:
  87. print(f"[finish] {code}: no arena ({exc})"); sys.exit(0)
  88. for path in sorted(glob.glob(f"{root}/{code}/.sapper/facts/*.json")):
  89. try:
  90. fact = json.load(open(path))
  91. except Exception:
  92. continue
  93. if fact.get("kind") != "flag":
  94. continue
  95. data = fact.get("data") or {}
  96. if data.get("status") not in (None, "recorded") or not data.get("value"):
  97. continue
  98. try:
  99. r = client.submit_flag(code, data["value"])
  100. fact["data"]["status"] = "submitted" if r.get("correct") else "rejected"
  101. json.dump(fact, open(path, "w"), ensure_ascii=False, indent=2)
  102. print(f"[finish] {code}: submit {data['value'][:24]}… correct={r.get('correct')}")
  103. except Exception as exc:
  104. msg = str(exc)
  105. if "already" in msg.lower() or "409" in msg:
  106. # duplicate submit = the flag WAS accepted earlier: absorb it
  107. fact["data"]["status"] = "submitted"
  108. json.dump(fact, open(path, "w"), ensure_ascii=False, indent=2)
  109. print(f"[finish] {code}: already submitted — marked ({data['value'][:20]}…)")
  110. else:
  111. print(f"[finish] {code}: submit failed ({exc})")
  112. PYFIN
  113. }
  114. # ---- pane primitives (verbatim from tmux-drive.sh) ---------------------------
  115. pane_text() { tmux capture-pane -pt "$SESSION:$1" 2>/dev/null; }
  116. send_literal() { # send_literal <window> <text> [enter] — NEVER bracketed paste
  117. tmux send-keys -t "$SESSION:$1" -l -- "$2"
  118. sleep 0.5
  119. [[ "${3:-enter}" == "enter" ]] && tmux send-keys -t "$SESSION:$1" Enter
  120. }
  121. accept_dialogs() { # accept_dialogs <window> <pane-text> -> 0 if a dialog was dismissed
  122. local win="$1" pane="$2"
  123. if printf '%s' "$pane" | grep -qE 'Enter to confirm|Yes, I accept|I accept'; then
  124. # Aim by the ❯ focus marker: security dialogs default-focus "No, exit",
  125. # so a bare Enter CHOOSES NO. Only move+confirm when focus is off Yes.
  126. if printf '%s' "$pane" | grep -qE '❯[^\n]*No, exit'; then
  127. tmux send-keys -t "$SESSION:$win" Down; sleep 0.4
  128. fi
  129. tmux send-keys -t "$SESSION:$win" Enter; sleep 1.5
  130. return 0
  131. fi
  132. return 1
  133. }
  134. # ---- worker-pane registry ----------------------------------------------------
  135. worker_windows() { # every window except the commander's window 0
  136. tmux list-windows -t "$SESSION" -F '#W' 2>/dev/null | grep -vx commander || true
  137. }
  138. count_workers() {
  139. local n; n=$(worker_windows | grep -c . || true)
  140. printf '%s' "${n:-0}"
  141. }
  142. panes_for_code() { # panes_for_code <code>
  143. local re=${1//./\\.}
  144. worker_windows | grep -E "^${re}-w[0-9]+$" || true
  145. }
  146. sync_session_env() { # tmux new-window inherits the SESSION environment, not the
  147. # calling client's env (only update-environment vars are
  148. # copied). Push the live gateway/platform env in so a key
  149. # rotated by the watchdog reaches freshly spawned panes;
  150. # PATH so the panes see the same tooling as the commander.
  151. local k v
  152. for k in PATH ANTHROPIC_BASE_URL ANTHROPIC_AUTH_TOKEN SAPPER_MODEL \
  153. SAPPER_MODEL_FLASH BENCHMARK_BASE_URL BENCHMARK_TOKEN; do
  154. v=${!k:-}
  155. [[ -n "$v" ]] && tmux set-environment -t "$SESSION" "$k" "$v" >/dev/null 2>&1 || true
  156. done
  157. }
  158. # ---- activity tracking (worker.sh is one-shot; state lives on disk) ----------
  159. declare -A OLD_TS OLD_SHA NEW_TS NEW_SHA
  160. collect_panes() { # prints TSV: name <TAB> alive|exited <TAB> idle <TAB> since <TAB> WxH
  161. OLD_TS=(); OLD_SHA=(); NEW_TS=(); NEW_SHA=()
  162. local w ts h text st since now; now=$(NOW)
  163. if [[ -f "$ACT_FILE" ]]; then
  164. while IFS=$'\t' read -r w ts h; do OLD_TS[$w]=$ts; OLD_SHA[$w]=$h; done < "$ACT_FILE"
  165. fi
  166. while IFS='|' read -r w created cols rows; do
  167. [[ -n "$w" ]] || continue
  168. [[ "$w" == "commander" ]] && continue # window 0 is the commander, not a worker
  169. text=$(pane_text "$w")
  170. if printf '%s' "$text" | grep -q 'DRIVE_PANE_EXITED'; then st=exited; else st=alive; fi
  171. h=$(printf '%s' "$text" | sha256sum | cut -c1-16)
  172. if [[ "${OLD_SHA[$w]:-}" == "$h" && -n "${OLD_TS[$w]:-}" ]]; then ts=${OLD_TS[$w]}; else ts=$now; fi
  173. NEW_TS[$w]=$ts; NEW_SHA[$w]=$h
  174. since=$(date -d "@${created:-$now}" +%H:%M 2>/dev/null || echo '?')
  175. printf '%s\t%s\t%s\t%s\t%sx%s\n' "$w" "$st" "$(( now - ts ))" "$since" "${cols:-?}" "${rows:-?}"
  176. done < <(tmux list-windows -t "$SESSION" -F '#{window_name}|#{window_created}|#{window_width}|#{window_height}' 2>/dev/null)
  177. local tmp; tmp=$(mktemp) || return 0
  178. : > "$tmp"
  179. for w in "${!NEW_TS[@]}"; do
  180. printf '%s\t%s\t%s\n' "$w" "${NEW_TS[$w]}" "${NEW_SHA[$w]}" >> "$tmp"
  181. done
  182. mv "$tmp" "$ACT_FILE" # atomic; entries for dead panes are pruned
  183. }
  184. # ---- trust preset (verbatim from tmux-drive.sh launch()) ----------------------
  185. trust_preset() { # trust_preset <dir>
  186. local dir=$1
  187. mkdir -p "$dir/.claude"
  188. printf '%s\n' '{"permissions":{"defaultMode":"bypassPermissions"},"autoCompactEnabled":true}' > "$dir/.claude/settings.json"
  189. python3 - "$dir" <<'PYTRUST'
  190. import json, os, sys
  191. path = os.path.expanduser("~/.claude.json")
  192. try: d = json.load(open(path))
  193. except Exception: d = {}
  194. d.setdefault("projects", {})[sys.argv[1]] = {
  195. "hasTrustDialogAccepted": True, "bypassPermissionsModeAccepted": True}
  196. json.dump(d, open(path, "w"))
  197. PYTRUST
  198. }
  199. ensure_graph() { # ensure_graph <code> — guarded sapper init in the primary dir
  200. local code=$1
  201. [[ -f "$ROOT/$code/.sapper/project.json" ]] && return 0
  202. mkdir -p "$ROOT/$code"
  203. # $SAPPER is intentionally unquoted: SAPPER_BIN may carry its own args
  204. (cd "$ROOT/$code" && $SAPPER init --goal "solve $code, capture all flags" >/dev/null 2>&1 || true)
  205. }
  206. write_hint() { # write_hint <code> <text> — appends a [commander]-prefixed hint fact
  207. local code=$1 text=$2 store="$ROOT/$1/.sapper" hid
  208. ensure_graph "$code"
  209. hid=$($SAPPER --store "$store" hint "[commander] $text" --creator commander 2>&1) \
  210. || die 1 "hint: sapper hint failed for $code: $hid"
  211. log "hint -> $code: $hid (${text:0:50})"
  212. printf 'HINT %s -> %s\n' "$hid" "$code"
  213. }
  214. close_instance() { # close_instance <code> — finish_challenge pattern: flags first,
  215. # then arena close (guarded: a 409/HTTP error never crashes)
  216. local code=$1
  217. log "$code: submitting recorded flags before close"
  218. finish_flags "$code" 2>&1 | tee -a "$LOG" >&2 || true
  219. if arena close "$code" >/dev/null 2>&1; then
  220. log "$code: platform instance closed"
  221. else
  222. log "$code: arena close failed (instance may already be closed) — continuing"
  223. fi
  224. rm -f "$ROOT/$code/.addrs"
  225. }
  226. general_mission() { # general_mission <code> <addrs> — smart worker allowed to hire -p squads
  227. local code=$1 addrs=$2
  228. printf '%s' "You are a GENERAL (senior operator) on challenge $code. Think DEEPLY — you have the biggest brain on this challenge, use it: plan the full attack chain before executing, reason step by step. Targets: ${addrs}. You solve the challenge yourself AND you may hire one-shot squads for bounded chores: run /opt/sapper/docker/commander/worker.sh spawn $code --p --minutes 5 --mission \"<chore: recon sweep / verify a guess / fuzz one endpoint>\" — the squad runs ~5 min, prints results in its pane, then goes idle: peek its pane (worker.sh peek <pane> 120), harvest, and kill it (worker.sh kill <pane>). Squads share this challenge's fact graph — tell them to read it first. Caps: panes are limited (a spawn failing with exit 4 means wait or fire a squad); keep at most 2 squads at a time; never let a squad idle long after it finishes. Record findings as facts, capture EVERY flag and IMMEDIATELY run sapper flag \"flag{...}\" -c $code --submit. Communication with your commander: [HELP]/[REPORT] notes in the graph as usual. Work fully autonomously; reply DONE only when the challenge is finished. Flag submission: primary is sapper flag VALUE -c $code --submit; if that command errors, submit by direct POST to the platform endpoint $BENCHMARK_BASE_URL/openapi/v1/challenges/submit with a JSON body containing unique_code and flag fields plus header BENCHMARK_TOKEN from env — write the one-liner yourself with urllib or curl; both routes count."
  229. }
  230. default_mission() { # default_mission <code> <addrs> <primary 0|1>
  231. # operator prompt carried over from tmux-drive.sh launch()
  232. local code=$1 addrs=$2 primary=$3 extra=""
  233. (( ! primary )) && extra=" Another agent is also working $code from a sibling directory — you SHARE one fact graph: read the existing facts before repeating work, and record every new finding with sapper fact so your partner sees it."
  234. printf '%s' "You are an autonomous pentest operator in a CTF benchmark. Think DEEPLY before acting: plan multi-step exploits carefully, reason about what each command's output tells you before the next one — depth of thought is your main weapon. Challenge: $code. Target container addresses (reach them directly): ${addrs}. Mission: recon the target, find the vulnerabilities, capture EVERY flag, and for each flag IMMEDIATELY run: sapper flag \"flag{...}\" -c $code --submit (this submits to the scoring platform and prints the result). Record findings as typed facts: sapper fact \"...\" -k service -d host=... -d port=... Consult the knowledge base when unsure: sapper kb search \"keywords\". Multiple flags are likely — after one submission keep hunting the rest. Communication: your commander reads this challenge's fact graph every ~2 minutes — to reach it, record a fact: sapper fact \"[HELP] <exactly what you tried, what blocks you, what decision you need>\" -k note (only when GENUINELY blocked, max one open HELP at a time; keep working alternatives while waiting) or \"[REPORT] <significant intel>\" -k note (interesting findings your commander may relay to other challenges). Do not wait idle for replies. When totally done, reply with the single word DONE.$extra"
  235. }
  236. # ---- spawn -------------------------------------------------------------------
  237. cmd_spawn() {
  238. local code="" mode=tui mission="" minutes=$DEF_MINUTES name="" model="" gw="" engine=claude general=0 a
  239. while (( $# )); do
  240. a=$1; shift
  241. case "$a" in
  242. --tui) mode=tui ;;
  243. --p|-p) mode=p ;;
  244. --general) general=1 ;;
  245. --gw) [[ $# -gt 0 ]] || die 1 "spawn: --gw needs a value (glm|ds)"; gw=$1; shift ;;
  246. --gw=*) gw=${a#*=} ;;
  247. --engine) [[ $# -gt 0 ]] || die 1 "spawn: --engine needs a value (claude|codex)"; engine=$1; shift ;;
  248. --engine=*) engine=${a#*=} ;;
  249. --mission) [[ $# -gt 0 ]] || die 1 "spawn: --mission needs a value"; mission=$1; shift ;;
  250. --mission=*) mission=${a#*=} ;;
  251. --minutes) [[ $# -gt 0 ]] || die 1 "spawn: --minutes needs a value"; minutes=$1; shift ;;
  252. --minutes=*) minutes=${a#*=} ;;
  253. --name) [[ $# -gt 0 ]] || die 1 "spawn: --name needs a value"; name=$1; shift ;;
  254. --name=*) name=${a#*=} ;;
  255. --model) [[ $# -gt 0 ]] || die 1 "spawn: --model needs a value"; model=$1; shift ;;
  256. --model=*) model=${a#*=} ;;
  257. -*) die 1 "spawn: unknown option $a" ;;
  258. *) [[ -z "$code" ]] || die 1 "spawn: unexpected argument $a"; code=$a ;;
  259. esac
  260. done
  261. [[ -z "$gw" || "$gw" == glm || "$gw" == ds ]] || die 1 "spawn: --gw must be glm or ds (got '$gw')"
  262. [[ "$engine" == claude || "$engine" == codex ]] || die 1 "spawn: --engine must be claude or codex"
  263. [[ -n "$code" ]] || die 1 "usage: worker.sh spawn <code> [--tui|--p] [--mission \"...\"] [--minutes N] [--name NAME] [--model <id>] [--gw glm|ds] [--engine claude|codex]"
  264. [[ "$gw" == ds && -z "$DS_KEY" && "$engine" == claude ]] \
  265. && die 1 "spawn: --gw ds needs DEEPSEEK_API_KEY (unset — ask the operator, do not retry)"
  266. # default model per gateway: glm->flash tier, ds->deepseek-flash;
  267. # a GENERAL defaults to the smart tier (SKILL 4.10). NOTE: compare with
  268. # quoted `==` — unquoted/glob paths treat the [1m] suffix as a char class
  269. # and silently never match.
  270. [[ -n "$model" ]] || { [[ "$gw" == ds ]] && model="$DS_MODEL" || model="$MODEL"; }
  271. if (( general )) && [[ "$gw" == glm && "$model" == "$MODEL" ]]; then
  272. model="${SAPPER_MODEL:-glm-5.3[1m]}"
  273. fi
  274. # gateway auto-derives from the FINAL model's family (mixed doctrine: the
  275. # default soldier model is deepseek — its name on the glm gateway 404s,
  276. # which killed every soldier pane in a live run). Explicit --gw wins.
  277. [[ -z "$gw" ]] && { case "$model" in *deepseek*) gw=ds;; *) gw=glm;; esac; }
  278. is_code "$code" || die 1 "spawn: invalid code '$code'"
  279. [[ "$minutes" =~ ^[0-9]+$ ]] || die 1 "spawn: --minutes must be an integer (got '$minutes')"
  280. tmux has-session -t "$SESSION" 2>/dev/null \
  281. || die 1 "tmux session '$SESSION' missing (entrypoint/watchdog owns it)"
  282. # hard worker cap BEFORE touching the platform (an instance slot is precious)
  283. local nw; nw=$(count_workers)
  284. if (( nw >= MAX_WORKERS )); then
  285. die 4 "WORKER CAP: $nw/$MAX_WORKERS worker panes live — kill one first (worker.sh kill <pane>)"
  286. fi
  287. local existing; existing=$(panes_for_code "$code")
  288. local max=0 n w
  289. for w in $existing; do
  290. [[ "$w" =~ ^${code//./\\.}-w([0-9]+)$ ]] && { n=${BASH_REMATCH[1]}; (( n > max )) && max=$n; }
  291. done
  292. local ordinal=$(( max + 1 ))
  293. local pane
  294. if [[ -n "$name" ]]; then
  295. pane=$name
  296. [[ "$pane" =~ ^[A-Za-z0-9_-]+$ ]] || die 1 "spawn: --name must be [A-Za-z0-9_-] (tmux target parsing)"
  297. if window_exists "$pane"; then
  298. # re-spawning one of THIS code's own worker panes is allowed (idempotent);
  299. # colliding with a foreign window is not
  300. [[ "$pane" =~ ^${code//./\\.}-w[0-9]+$ ]] \
  301. || die 1 "spawn: window '$pane' already exists (belongs to another pane)"
  302. fi
  303. else
  304. pane="${code}-w${ordinal}"
  305. fi
  306. # primary agent = first pane of this code; it owns ROOT/<code>; -w2+ get their
  307. # own dir with the shared graph symlinked in (no same-cwd --continue crosstalk)
  308. local primary=0 dir="$ROOT/$code"
  309. (( max == 0 )) && primary=1
  310. (( primary )) || dir="$ROOT/${code}-w${ordinal}"
  311. # 1) platform instance — only when this code has no live pane and no known addrs
  312. local addrs="" started="" got="" err
  313. [[ -f "$ROOT/$code/.addrs" ]] && addrs=$(cat "$ROOT/$code/.addrs" 2>/dev/null)
  314. if (( ! primary )) && [[ -n "$addrs" ]]; then
  315. : # instance already up (shared with w1) — do not re-start, just reuse addrs
  316. else
  317. err=$(mktemp)
  318. started=$(arena start "$code" 2>"$err" || true)
  319. got=$(printf '%s' "$started" | python3 -c '
  320. import json,sys
  321. try: print(" ".join(json.load(sys.stdin).get("container_addr") or []))
  322. except Exception: pass' 2>/dev/null)
  323. if [[ -n "$got" ]]; then
  324. addrs=$got
  325. printf '%s\n' "$addrs" > "$ROOT/$code/.addrs"
  326. elif grep -qi "max active" "$err" 2>/dev/null; then
  327. rm -f "$err"
  328. die 3 "PLATFORM CAP: close a challenge first (worker.sh kill <code> --close) — cannot start $code"
  329. elif [[ -z "$addrs" ]]; then
  330. local e; e=$(tail -n 3 "$err" 2>/dev/null | tr '\n' ' ')
  331. rm -f "$err"
  332. die 1 "spawn: arena start failed for $code: ${e:-no container_addr returned}"
  333. else
  334. log "$code: arena start refused but persisted addrs look valid — reusing instance"
  335. fi
  336. rm -f "$err"
  337. fi
  338. # 2) dirs + shared graph
  339. if (( primary )); then
  340. mkdir -p "$dir"
  341. ensure_graph "$code"
  342. else
  343. ensure_graph "$code"
  344. mkdir -p "$dir"
  345. ln -sfn "../$code/.sapper" "$dir/.sapper"
  346. fi
  347. # 3) trust preset
  348. trust_preset "$dir"
  349. # 4/5) the window (NO --continue: fresh session per pane; watchdog owns recovery)
  350. sync_session_env
  351. tmux kill-window -t "$SESSION:$pane" 2>/dev/null
  352. # persist the pane's tier+gateway so watchdog relaunches keep the same brain.
  353. # RAW values, never %q: bash %q escapes the [1m] suffix -> '\\[1m\\' and
  354. # pane_model()'s sed would feed claude a backslashed model id. TUI missions
  355. # are archived base64 so the watchdog can re-brief a restarted pane (a
  356. # relaunched claude is a FRESH session — nobody else will send its mission).
  357. local mission_b64=""
  358. [[ "$mode" == tui && "$engine" == claude && -n "$mission" ]] \
  359. && mission_b64=$(printf '%s' "$mission" | base64 -w0)
  360. printf 'MODEL=%s\nGW=%s\nENGINE=%s\nMISSION_B64=%s\n' \
  361. "$model" "$gw" "$engine" "$mission_b64" > "$PANE_ENV_DIR/$pane.env"
  362. # per-pane gateway env: ds panes override ANTHROPIC_* with the deepseek endpoint
  363. local gw_env=""
  364. [[ "$gw" == ds ]] && gw_env="ANTHROPIC_BASE_URL='$DS_BASE_URL' ANTHROPIC_AUTH_TOKEN='$DS_KEY'"
  365. if [[ "$mode" == tui ]]; then
  366. if [[ "$engine" == codex ]]; then
  367. # codex 0.42: provider via -c overrides on top of ~/.codex/config.toml
  368. # (chat wire for DeepSeek). ds-local = official endpoint for local sims.
  369. local provider=deepseek; [[ "$DS_BASE_URL" == https://api.deepseek.com* ]] && provider=deepseek-local
  370. tmux new-window -d -t "$SESSION" -n "$pane" \
  371. "cd $dir && DEEPSEEK_API_KEY='$DS_KEY' codex --skip-git-repo-check -c model_provider=$provider -c model='$model' --dangerously-bypass-approvals-and-sandbox 2>&1; echo DRIVE_PANE_EXITED; exec sleep infinity"
  372. else
  373. tmux new-window -d -t "$SESSION" -n "$pane" \
  374. "cd $dir && $gw_env claude --model '$model' --effort max --dangerously-skip-permissions 2>&1; echo DRIVE_PANE_EXITED; exec sleep infinity"
  375. fi
  376. else
  377. if [[ "$engine" == codex ]]; then
  378. [[ -n "$mission" ]] || mission=$(default_mission "$code" "$addrs" "$primary")
  379. local esc_mission=${mission//\'/\\'\'}
  380. tmux new-window -d -t "$SESSION" -n "$pane" \
  381. "cd $dir && DEEPSEEK_API_KEY='$DS_KEY' codex exec --skip-git-repo-check -c model_provider=deepseek-local -c model='$model' --dangerously-bypass-approvals-and-sandbox '$esc_mission' 2>&1; echo DRIVE_PANE_EXITED; exec sleep infinity"
  382. printf 'SPAWNED %s addrs=%s model=%s engine=codex\n' "$pane" "$addrs" "$model"
  383. return 0
  384. else
  385. tmux new-window -d -t "$SESSION" -n "$pane" \
  386. "cd $dir && $gw_env SAPPER_MODEL='$model' $SAPPER --store $ROOT/$code/.sapper auto --minutes $minutes --skip-permissions 2>&1; echo DRIVE_PANE_EXITED; exec sleep infinity"
  387. fi
  388. fi
  389. # 6) mission
  390. if [[ "$mode" == tui && "$engine" == codex ]]; then
  391. # codex TUI readiness: no claude-style ❯ marker; wait for the process to
  392. # settle, then type the mission (codex accepts typed input at its prompt)
  393. sleep 12
  394. [[ -z "$mission" ]] && mission=$(default_mission "$code" "$addrs" "$primary")
  395. send_literal "$pane" "$mission"
  396. log "$pane: codex TUI spawned (dir=$dir), mission sent"
  397. elif [[ "$mode" == tui ]]; then
  398. local waited=0 ptext="" frag
  399. while (( waited < 75 )); do
  400. ptext=$(pane_text "$pane")
  401. accept_dialogs "$pane" "$ptext" && { waited=$((waited+3)); continue; }
  402. printf '%s' "$ptext" | grep -q '❯' && break
  403. sleep 1; waited=$((waited+1))
  404. done
  405. printf '%s' "$ptext" | grep -q '❯' \
  406. || die 1 "spawn: $pane input box never appeared after 75s (pane left for inspection)"
  407. if [[ -z "$mission" ]]; then
  408. if (( general )); then mission=$(general_mission "$code" "$addrs")
  409. else mission=$(default_mission "$code" "$addrs" "$primary"); fi
  410. fi
  411. send_literal "$pane" "$mission"
  412. sleep 3
  413. ptext=$(pane_text "$pane")
  414. frag=${mission:0:24}
  415. if ! printf '%s' "$ptext" | grep -qF -- "$frag" \
  416. && ! printf '%s' "$ptext" | grep -q 'esc to interrupt'; then
  417. log "$pane: mission not confirmed in pane — retrying once"
  418. send_literal "$pane" "$mission"
  419. sleep 3
  420. fi
  421. log "$pane: TUI spawned (dir=$dir), mission sent"
  422. else
  423. sleep 2
  424. if printf '%s' "$(pane_text "$pane")" | grep -q 'DRIVE_PANE_EXITED'; then
  425. die 1 "spawn: $pane -p command exited immediately — check the sapper invocation in the pane"
  426. fi
  427. [[ -n "$mission" ]] && write_hint "$code" "$mission"
  428. log "$pane: -p worker spawned (dir=$dir, minutes=$minutes)"
  429. fi
  430. # 7)
  431. printf 'SPAWNED %s addrs=%s model=%s\n' "$pane" "$addrs" "$model"
  432. }
  433. # ---- kill --------------------------------------------------------------------
  434. cmd_kill() {
  435. local close=0 target="" a
  436. while (( $# )); do
  437. a=$1; shift
  438. case "$a" in
  439. --close) close=1 ;;
  440. -*) die 1 "kill: unknown option $a" ;;
  441. *) [[ -z "$target" ]] || die 1 "kill: unexpected argument $a"; target=$a ;;
  442. esac
  443. done
  444. [[ -n "$target" ]] || die 1 "usage: worker.sh kill <pane>|<code> [--close]"
  445. tmux has-session -t "$SESSION" 2>/dev/null \
  446. || die 1 "tmux session '$SESSION' missing (entrypoint/watchdog owns it)"
  447. local re=${target//./\\.} wins
  448. if window_exists "$target"; then
  449. wins=$target
  450. else
  451. wins=$(worker_windows | grep -E "^${re}-w[0-9]+$" || true)
  452. fi
  453. [[ -n "$wins" ]] || die 1 "kill: no pane matches '$target'"
  454. local code=$target w
  455. [[ "$code" =~ -w[0-9]+$ ]] && code=${code%-w*}
  456. for w in $wins; do
  457. if tmux kill-window -t "$SESSION:$w" 2>/dev/null; then
  458. log "killed $w"
  459. else
  460. log "kill: window $w already gone"
  461. fi
  462. rm -f "$PANE_ENV_DIR/$w.env"
  463. done
  464. if (( close )); then
  465. local left; left=$(panes_for_code "$code")
  466. if [[ -n "$left" ]]; then
  467. log "$code: sibling panes still alive ($left) — instance kept open, flags not touched"
  468. return 0
  469. fi
  470. close_instance "$code"
  471. fi
  472. }
  473. # ---- msg / peek / list -------------------------------------------------------
  474. cmd_msg() {
  475. local pane=${1:-} text=${2:-}
  476. [[ -n "$pane" && -n "$text" ]] || die 1 'usage: worker.sh msg <pane> "<text>"'
  477. tmux has-session -t "$SESSION" 2>/dev/null || die 1 "tmux session '$SESSION' missing"
  478. window_exists "$pane" || die 1 "msg: no such pane '$pane'"
  479. send_literal "$pane" "$text"
  480. log "msg -> $pane: ${text:0:60}"
  481. }
  482. cmd_peek() {
  483. local pane=${1:-} lines=${2:-40}
  484. [[ -n "$pane" ]] || die 1 "usage: worker.sh peek <pane> [lines=40]"
  485. [[ "$lines" =~ ^[0-9]+$ ]] || die 1 "peek: lines must be an integer"
  486. tmux has-session -t "$SESSION" 2>/dev/null || die 1 "tmux session '$SESSION' missing"
  487. window_exists "$pane" || die 1 "peek: no such pane '$pane'"
  488. tmux capture-pane -p -t "$SESSION:$pane" 2>/dev/null | tail -n "$lines"
  489. }
  490. cmd_list() {
  491. local info; info=$(collect_panes) || true
  492. if [[ -z "$info" ]]; then
  493. printf '(no worker panes)\n'
  494. return 0
  495. fi
  496. local w st idle since size
  497. while IFS=$'\t' read -r w st idle since size; do
  498. printf '%s %s idle=%ss size=%s since=%s\n' "$w" "$st" "$idle" "$size" "$since"
  499. done <<< "$info"
  500. }
  501. # ---- board -------------------------------------------------------------------
  502. cmd_board() {
  503. local as_json=0 a
  504. while (( $# )); do
  505. a=$1; shift
  506. case "$a" in
  507. --json) as_json=1 ;;
  508. *) die 1 "board: unknown option $a (try --json)" ;;
  509. esac
  510. done
  511. local arena_tmp pane_tmp rc
  512. arena_tmp=$(mktemp); pane_tmp=$(mktemp)
  513. arena list > "$arena_tmp" 2>/dev/null || printf '[]\n' > "$arena_tmp"
  514. collect_panes > "$pane_tmp" || true
  515. python3 - "$arena_tmp" "$pane_tmp" "$ROOT" "$MAX_WORKERS" "$as_json" <<'PYBOARD'
  516. import glob, json, os, re, sys, time
  517. arena_path, pane_path, root = sys.argv[1], sys.argv[2], sys.argv[3]
  518. max_workers, as_json = int(sys.argv[4]), sys.argv[5] == "1"
  519. def load_json(path, default):
  520. try:
  521. with open(path) as fh:
  522. return json.load(fh)
  523. except Exception:
  524. return default
  525. # -- platform rows (arena list; unreachable platform degrades to empty) --------
  526. plat = {}
  527. for c in (load_json(arena_path, []) or []):
  528. if isinstance(c, dict) and isinstance(c.get("unique_code"), str):
  529. plat[c["unique_code"]] = {
  530. "completed": bool(c.get("is_completed")),
  531. "flags_correct": int(c.get("correct_flag_count") or 0),
  532. "flags_total": int(c.get("flag_count") or 0),
  533. }
  534. # -- graphs (dedupe -wN symlinked graphs by realpath; never trust file contents)
  535. graphs = {}
  536. for pj in sorted(glob.glob(os.path.join(root, "*", ".sapper", "project.json"))):
  537. d = os.path.dirname(os.path.dirname(pj))
  538. code = re.sub(r"-w\d+$", "", os.path.basename(d))
  539. key = os.path.realpath(os.path.join(d, ".sapper"))
  540. if key in graphs:
  541. continue
  542. kinds, latest, total = {}, 0.0, 0
  543. flags = {"recorded": 0, "submitted": 0}
  544. helps = 0
  545. for fp in glob.glob(os.path.join(d, ".sapper", "facts", "*.json")):
  546. total += 1
  547. try:
  548. f = json.load(open(fp))
  549. latest = max(latest, os.path.getmtime(fp))
  550. except Exception:
  551. continue
  552. k = f.get("kind")
  553. if k:
  554. kinds[k] = kinds.get(k, 0) + 1
  555. if k == "flag":
  556. st = (f.get("data") or {}).get("status") or "recorded"
  557. flags[st] = flags.get(st, 0) + 1
  558. if k == "note" and "[HELP]" in str(f.get("description", "")) + str(f.get("data") or ""):
  559. helps += 1
  560. intents = {"open": 0, "claimed": 0, "done": 0, "complete": 0}
  561. for ip in glob.glob(os.path.join(d, ".sapper", "intents", "*.json")):
  562. try:
  563. st = json.load(open(ip)).get("status")
  564. except Exception:
  565. continue
  566. if st in intents:
  567. intents[st] += 1
  568. graphs[key] = {"code": code, "facts_total": total, "kinds": kinds,
  569. "flags_recorded": flags["recorded"],
  570. "flags_submitted": flags["submitted"],
  571. "intents": intents, "latest_mtime": latest, "helps": helps}
  572. by_code = {}
  573. for g in graphs.values():
  574. cur = by_code.get(g["code"])
  575. if cur is None or g["facts_total"] > cur["facts_total"]:
  576. by_code[g["code"]] = g
  577. # -- panes merged in ------------------------------------------------------------
  578. panes, alive = {}, 0
  579. for line in open(pane_path):
  580. parts = line.rstrip("\n").split("\t")
  581. if len(parts) != 5:
  582. continue
  583. w, st, idle, since, size = parts
  584. m = re.match(r"^(.*)-w\d+$", w)
  585. panes.setdefault(m.group(1) if m else w, []).append(
  586. {"pane": w, "state": st, "idle": int(idle), "since": since, "size": size})
  587. if st == "alive":
  588. alive += 1
  589. codes = sorted(set(plat) | set(by_code) | set(panes),
  590. key=lambda c: (0 if c in panes else (1 if c in by_code else 2), c))
  591. rows = []
  592. for c in codes:
  593. p, g = plat.get(c) or {}, by_code.get(c) or {}
  594. rows.append({
  595. "code": c,
  596. "completed": p.get("completed"),
  597. "platform_flags": ([p["flags_correct"], p["flags_total"]] if p else None),
  598. "facts_total": g.get("facts_total", 0),
  599. "kinds": g.get("kinds", {}),
  600. "intents": g.get("intents", {}),
  601. "flags_recorded": g.get("flags_recorded", 0),
  602. "flags_submitted": g.get("flags_submitted", 0),
  603. "helps": g.get("helps", 0),
  604. "latest": time.strftime("%H:%M", time.localtime(g["latest_mtime"]))
  605. if g.get("latest_mtime") else "-",
  606. "panes": panes.get(c, []),
  607. })
  608. if as_json:
  609. print(json.dumps({"now": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
  610. "workers_alive": alive, "worker_cap": max_workers,
  611. "codes": rows}, ensure_ascii=False, separators=(",", ":")))
  612. raise SystemExit(0)
  613. # heartbeat to drive.log (=> docker logs): the commander boards every loop,
  614. # so this line IS the campaign pulse for the outside observer.
  615. flags_r = sum(r["flags_recorded"] for r in rows)
  616. flags_s = sum(r["flags_submitted"] for r in rows)
  617. done_n = sum(1 for r in rows if r["completed"])
  618. # war clock + general census: the commander's dashboard gauges
  619. left = "-"
  620. try:
  621. dl = float(open(os.path.join(root, ".deadline")).read().strip())
  622. secs = dl - time.time()
  623. if secs >= 0:
  624. left = ("%dh%02dm" % (secs // 3600, (secs % 3600) // 60))
  625. else:
  626. left = "OVER"
  627. except Exception:
  628. pass
  629. generals = 0
  630. try:
  631. import glob as _g
  632. for pf in _g.glob(os.path.join(root, "pane-env", "*.env")):
  633. try:
  634. m = [l for l in open(pf) if l.startswith("MODEL=")][0][6:].strip()
  635. except Exception:
  636. continue
  637. if m and "flash" not in m and "deepseek" not in m:
  638. generals += 1
  639. except Exception:
  640. pass
  641. print(f"[board] heartbeat: codes={len(rows)} done={done_n} "
  642. f"alive={alive}/{max_workers} flags_recorded={flags_r} "
  643. f"flags_submitted={flags_s} budget_left={left} generals={generals}", file=sys.stderr)
  644. def kinds_str(k):
  645. if not k:
  646. return "-"
  647. order = ["host", "service", "cred", "vuln", "loot", "flag", "note"]
  648. parts = [f"{n[0]}{k[n]}" for n in order if k.get(n)]
  649. parts += [f"{n[:2]}{v}" for n, v in sorted(k.items()) if n not in order]
  650. return (" ".join(parts) or "-")[:16]
  651. def panes_str(lst):
  652. if not lst:
  653. return "-"
  654. out = []
  655. for p in lst:
  656. m = re.match(r"^.*-w(\d+)$", p["pane"])
  657. lbl = f"w{m.group(1)}" if m else p["pane"]
  658. out.append(f"{lbl}:{p['idle']}s" if p["state"] == "alive" else f"{lbl}:X")
  659. return " ".join(out)
  660. widths = [("CODE(⚑=HELP!)", 17), ("DONE", 4), ("FLG", 7), ("FACTS", 6), ("KINDS", 16),
  661. ("INT o/c", 8), ("FLG r/s", 7), ("LAST", 5)]
  662. print(f"== sapper board {time.strftime('%H:%M')} workers alive={alive}/{max_workers}"
  663. f" codes={len(rows)} ==")
  664. print(" ".join(t.ljust(w) for t, w in widths) + "PANES")
  665. budget = 100 - 3 # header + column line + trailing guard
  666. for i, r in enumerate(rows):
  667. if i >= budget:
  668. print(f"... {len(rows) - i} more codes (use: worker.sh board --json)")
  669. break
  670. pf = r["platform_flags"]
  671. done = "-" if r["completed"] is None else ("yes" if r["completed"] else "no")
  672. codecell = r["code"][:14]
  673. if r.get("helps"):
  674. codecell = "⚑%d %s" % (r["helps"], codecell)[:14 + 3]
  675. cells = [
  676. codecell.ljust(17), done.ljust(4),
  677. (f"{pf[0]}/{pf[1]}" if pf else "-").ljust(7),
  678. (str(r["facts_total"]) if r["facts_total"] else "-").ljust(6),
  679. kinds_str(r["kinds"]).ljust(16),
  680. f"{r['intents'].get('open', 0)}/{r['intents'].get('claimed', 0)}".ljust(8),
  681. f"{r['flags_recorded']}/{r['flags_submitted']}".ljust(7),
  682. r["latest"].ljust(5), panes_str(r["panes"]),
  683. ]
  684. print(" ".join(cells).rstrip())
  685. PYBOARD
  686. rc=$?
  687. rm -f "$arena_tmp" "$pane_tmp"
  688. return "$rc"
  689. }
  690. # ---- hint --------------------------------------------------------------------
  691. cmd_hint() {
  692. local code=${1:-} text=${2:-}
  693. [[ -n "$code" && -n "$text" ]] || die 1 'usage: worker.sh hint <code> "<text>"'
  694. is_code "$code" || die 1 "hint: invalid code '$code'"
  695. write_hint "$code" "$text"
  696. }
  697. usage() {
  698. cat >&2 <<'USAGE'
  699. worker.sh — the commander's hands (one-shot subcommands; no main loop)
  700. spawn <code> [--tui|--p] [--mission "..."] [--minutes N] [--name NAME] [--model <id>]
  701. [--gw glm|ds] [--engine claude|codex] [--general]
  702. --model: worker brain (default flash tier; smart =
  703. 'glm-5.3[1m]' — SKILL 4.9 tiering rules)
  704. --general: smart-tier GENERAL that may hire its own
  705. -p squads on the same challenge (SKILL 4.10)
  706. kill <pane>|<code> [--close] --close submits flags + closes the instance
  707. only when no sibling pane of that code lives
  708. msg <pane> "<text>" send-keys -l + Enter (never bracketed paste)
  709. peek <pane> [lines=40]
  710. list <pane> alive|exited idle=<s> size=WxH since=<hh:mm>
  711. board [--json] the war board (<100 lines)
  712. hint <code> "<text>" append a [commander] hint fact to that graph
  713. exit codes: 0 ok | 1 generic error | 3 platform cap (max active) | 4 worker cap
  714. USAGE
  715. }
  716. case "${1:-}" in
  717. spawn) shift; cmd_spawn "$@" ;;
  718. kill) shift; cmd_kill "$@" ;;
  719. msg) shift; cmd_msg "$@" ;;
  720. peek) shift; cmd_peek "$@" ;;
  721. list) shift; cmd_list "$@" ;;
  722. board) shift; cmd_board "$@" ;;
  723. hint) shift; cmd_hint "$@" ;;
  724. help|-h|--help) usage; exit 0 ;;
  725. "") usage; exit 1 ;;
  726. *) printf '[worker] ERROR: unknown subcommand %s\n' "$1" >&2; usage; exit 1 ;;
  727. esac
docker/tmux-drive.sh · Bundled script · 363 lines · 1 finding marked
  1. #!/usr/bin/env bash
  2. # tmux-drive.sh v2 — self-healing driver for the TUI-mode agent fleet.
  3. #
  4. # Failure modes handled:
  5. # - Enter lost / mission stuck in the input box -> re-Enter (3 tries), then restart window
  6. # - trust/bypass/permission dialogs -> auto-accept (Down+Enter)
  7. # - claude TUI exits / pane dead -> relaunch window (max 2/challenge)
  8. # - agent stuck (no pane change) -> nudge; still stuck at budget -> stop
  9. # - driver restart -> adopts existing tmux windows (idempotent)
  10. # - challenge DONE or per-challenge timeout -> submit recorded flags, close, next
  11. set -o pipefail
  12. ROOT="${SAPPER_ROOT:-/work/runs}"
  13. LOG="$ROOT/drive.log"
  14. MODEL="${SAPPER_MODEL:-glm-5.3-flash[1m]}"
  15. PARALLEL="${SAPPER_PARALLEL:-6}"
  16. TOTAL_MIN="${SAPPER_TOTAL_MINUTES:-470}"
  17. PER_MIN="${SAPPER_MINUTES_PER_CHALLENGE:-25}"
  18. STALL_SECONDS="${SAPPER_STALL_SECONDS:-300}"
  19. SESSION=sapper-tui
  20. MAX_RESTARTS=2
  21. mkdir -p "$ROOT"
  22. log() { printf '[drive] %s\n' "$*" | tee -a "$LOG"; }
  23. END_AT=$(( $(date +%s) + TOTAL_MIN * 60 ))
  24. NOW() { date +%s; }
  25. declare -A launch_at last_activity restarts finished claim_start last_pane_state
  26. # ---- arena helper (graph-free) ----------------------------------------------
  27. arena() {
  28. python3 - "$@" <<'PYEOF2'
  29. import importlib.util, json, sys
  30. spec = importlib.util.spec_from_file_location("arena", "/opt/sapper/lib/arena.py")
  31. m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
  32. c = m.ArenaClient.load(None)
  33. op = sys.argv[1]
  34. if op == "list":
  35. print(json.dumps(c.list_challenges()))
  36. elif op == "start":
  37. print(json.dumps(c.start_challenge(sys.argv[2])))
  38. elif op == "close":
  39. print(json.dumps(c.close_challenge(sys.argv[2])))
  40. PYEOF2
  41. }
  42. # ---- key-pool rotation ------------------------------------------------------
  43. # SAPPER_KEY_POOL: JSON [{base_url,token,model},...]. On quota/balance/auth
  44. # errors the driver rotates its own env; windows relaunched afterwards
  45. # inherit the new key, and `claude --continue` restores the prior session.
  46. rotate_key() {
  47. python3 - <<'PYROT'
  48. import json, os
  49. raw = os.environ.get("SAPPER_KEY_POOL", "").strip()
  50. try:
  51. pool = [p for p in json.loads(raw) if isinstance(p, dict) and p.get("token")]
  52. except Exception:
  53. pool = []
  54. if not pool:
  55. raise SystemExit(1)
  56. cur = os.environ.get("ANTHROPIC_AUTH_TOKEN", "")
  57. start = next((i for i, e in enumerate(pool) if e["token"] == cur), -1)
  58. nxt = pool[(start + 1) % len(pool)]
  59. if start >= 0 and nxt["token"] == cur:
  60. raise SystemExit(1) # single key: nothing to rotate to
  61. for k, v in (
  62. ("ANTHROPIC_BASE_URL", nxt.get("base_url")),
  63. ("ANTHROPIC_AUTH_TOKEN", nxt["token"]),
  64. ("SAPPER_MODEL", nxt.get("model")),
  65. ):
  66. if v:
  67. os.environ[k] = v
  68. with open(os.environ.get("SAPPER_ROOT", "/work/runs") + "/rotated.env", "w") as fh:
  69. fh.write(
  70. f'export ANTHROPIC_BASE_URL="{os.environ["ANTHROPIC_BASE_URL"]}"\n'
  71. f'export ANTHROPIC_AUTH_TOKEN="{os.environ["ANTHROPIC_AUTH_TOKEN"]}"\n'
  72. f'export SAPPER_MODEL="{os.environ.get("SAPPER_MODEL", "")}"\n'
  73. )
  74. raise SystemExit(0)
  75. PYROT
  76. local rc=$?
  77. if (( rc == 0 )) && [[ -f "$ROOT/rotated.env" ]]; then
  78. # shellcheck disable=SC1090
  79. . "$ROOT/rotated.env"; export ANTHROPIC_BASE_URL ANTHROPIC_AUTH_TOKEN SAPPER_MODEL
  80. log "key rotated -> $ANTHROPIC_BASE_URL (model $SAPPER_MODEL)"
  81. return 0
  82. fi
  83. return 1
  84. }
  85. QUOTA_RE='insufficient|balance|quota|exceeded|billing|payment|401|[Uu]nauthorized|invalid.{0,12}[Aa][Pp][Ii].[Kk]ey|欠费|余额'
  86. # submit every recorded-unsubmitted flag fact in a challenge's graph
  87. finish_flags() { # finish_flags <code>
  88. python3 - "$1" <<'PYFIN'
  89. import glob, json, os, sys, importlib.util
  90. code = sys.argv[1]
  91. root = os.environ.get("SAPPER_ROOT", "/work/runs")
  92. spec = importlib.util.spec_from_file_location("arena", "/opt/sapper/lib/arena.py")
  93. m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
  94. try:
  95. client = m.ArenaClient.load(None)
  96. except Exception as exc:
  97. print(f"[finish] {code}: no arena ({exc})"); sys.exit(0)
  98. for path in sorted(glob.glob(f"{root}/{code}/.sapper/facts/*.json")):
  99. try:
  100. fact = json.load(open(path))
  101. except Exception:
  102. continue
  103. if fact.get("kind") != "flag":
  104. continue
  105. data = fact.get("data") or {}
  106. if data.get("status") not in (None, "recorded") or not data.get("value"):
  107. continue
  108. try:
  109. r = client.submit_flag(code, data["value"])
  110. fact["data"]["status"] = "submitted" if r.get("correct") else "rejected"
  111. json.dump(fact, open(path, "w"), ensure_ascii=False, indent=2)
  112. print(f"[finish] {code}: submit {data['value'][:24]}… correct={r.get('correct')}")
  113. except Exception as exc:
  114. print(f"[finish] {code}: submit failed ({exc})")
  115. PYFIN
  116. }
  117. finish_challenge() { # finish_challenge <code> <reason>
  118. local code="$1" reason="$2"
  119. [[ -n "${finished[$code]:-}" ]] && return 0
  120. finished[$code]=1
  121. log "$code: finishing ($reason)"
  122. finish_flags "$code" 2>&1 | tee -a "$LOG"
  123. arena close "$code" >/dev/null 2>&1 && log "$code: container closed"
  124. tmux kill-window -t "$SESSION:$code" 2>/dev/null
  125. }
  126. # ---- launch / adopt ---------------------------------------------------------
  127. pane_text() { tmux capture-pane -pt "$SESSION:$1" 2>/dev/null; }
  128. send_literal() { # send_literal <window> <text> [enter]
  129. tmux send-keys -t "$SESSION:$1" -l -- "$2"
  130. sleep 0.5
  131. [[ "${3:-enter}" == "enter" ]] && tmux send-keys -t "$SESSION:$1" Enter
  132. }
  133. accept_dialogs() { # accept_dialogs <window> <pane> -> 0 if a dialog was dismissed
  134. local win="$1" pane="$2"
  135. if printf '%s' "$pane" | grep -q 'Enter to confirm'; then
  136. tmux send-keys -t "$SESSION:$win" Down; sleep 0.4
  137. tmux send-keys -t "$SESSION:$win" Enter; sleep 1.5
  138. return 0
  139. fi
  140. return 1
  141. }
  142. launch() { # launch <code>
  143. local code="$1"
  144. local err; err=$(mktemp)
  145. local started; started=$(arena start "$code" 2>"$err" || true)
  146. LAST_LAUNCH_ERR="$(cat "$err" 2>/dev/null || true)"
  147. cat "$err" >>"$LOG" 2>/dev/null || true; rm -f "$err"
  148. local addrs; addrs=$(printf '%s' "$started" | python3 -c '
  149. import json,sys
  150. try: print(" ".join(json.load(sys.stdin).get("container_addr") or []))
  151. except Exception: pass' 2>/dev/null)
  152. log "$code: start -> ${addrs:-FAILED}"
  153. [[ -z "$addrs" ]] && return 1
  154. mkdir -p "$ROOT/$code/.claude"
  155. (cd "$ROOT/$code" && sapper init --goal "solve $code, capture all flags" >/dev/null 2>&1 || true)
  156. printf '%s\n' '{"permissions":{"defaultMode":"bypassPermissions"}}' > "$ROOT/$code/.claude/settings.json"
  157. python3 - "$ROOT/$code" <<'PYTRUST'
  158. import json, os, sys
  159. path = os.path.expanduser("~/.claude.json")
  160. try: d = json.load(open(path))
  161. except Exception: d = {}
  162. d.setdefault("projects", {})[sys.argv[1]] = {
  163. "hasTrustDialogAccepted": True, "bypassPermissionsModeAccepted": True}
  164. json.dump(d, open(path, "w"))
  165. PYTRUST
  166. tmux kill-window -t "$SESSION:$code" 2>/dev/null
  167. local resume_flag=""
  168. ls "$HOME/.claude/projects/"*"$code"*/ >/dev/null 2>&1 && resume_flag="--continue"
  169. tmux new-window -d -t "$SESSION" -n "$code" \
  170. "cd $ROOT/$code && claude $resume_flag --model '$MODEL' --dangerously-skip-permissions 2>&1; echo DRIVE_PANE_EXITED; exec sleep infinity"
  171. launch_at[$code]=$(NOW); last_activity[$code]=$(NOW)
  172. restarts[$code]=$(( ${restarts[$code]:-0} ))
  173. claim_start[$code]=0
  174. # wait for the input box, auto-accepting dialogs
  175. local waited=0 pane=""
  176. while (( waited < 75 )); do
  177. pane=$(pane_text "$code")
  178. accept_dialogs "$code" "$pane" && { waited=$((waited+3)); continue; }
  179. printf '%s' "$pane" | grep -q '❯' && break
  180. sleep 1; waited=$((waited+1))
  181. done
  182. printf '%s' "$pane" | grep -q '❯' || { log "$code: input box never appeared"; return 1; }
  183. local mission
  184. mission="You are an autonomous pentest operator in a CTF benchmark. Challenge: $code. Target container addresses (reach them directly): ${addrs}. Mission: recon the target, find the vulnerabilities, capture EVERY flag, and for each flag IMMEDIATELY run: sapper flag \"flag{...}\" -c $code --submit (this submits to the scoring platform and prints the result). Record findings as typed facts: sapper fact \"...\" -k service -d host=... -d port=... Consult the knowledge base when unsure: sapper kb search \"keywords\". Multiple flags are likely (${PER_MIN}min budget) — after one submission keep hunting the rest. Work fully autonomously, never ask questions. When totally done, reply with the single word DONE."
  185. send_literal "$code" "$mission"
  186. sleep 3
  187. # verify: mission text visible in the input box, OR agent already working
  188. pane=$(pane_text "$code")
  189. if ! printf '%s' "$pane" | grep -q 'autonomous pentest operator\|esc to interrupt'; then
  190. log "$code: mission not confirmed in pane — retrying once"
  191. send_literal "$code" "$mission"
  192. sleep 3
  193. fi
  194. last_activity[$code]=$(NOW)
  195. log "$code: TUI launched, mission sent"
  196. return 0
  197. }
  198. adopt_existing() {
  199. local win
  200. for win in $(tmux list-windows -t "$SESSION" -F '#W' 2>/dev/null | grep -v '^supervisor$'); do
  201. [[ -n "${launch_at[$win]:-}" ]] && continue
  202. launch_at[$win]=$(NOW); last_activity[$win]=$(NOW); restarts[$win]=0
  203. log "$win: adopted existing window after driver restart"
  204. done
  205. }
  206. list_pending() {
  207. arena list 2>>"$LOG" | python3 -c '
  208. import json,sys
  209. try: items=json.load(sys.stdin)
  210. except Exception: sys.exit(0)
  211. for c in items:
  212. if isinstance(c,dict) and not c.get("is_completed"): print(c["unique_code"])' 2>/dev/null
  213. }
  214. # ---- build the pending queue -------------------------------------------------
  215. mapfile -t ALL_CODES < <(list_pending)
  216. log "challenges pending: ${ALL_CODES[*]:-none} (parallel=$PARALLEL, per=$PER_MIN min, total=$TOTAL_MIN min)"
  217. idx=0
  218. adopt_existing
  219. # ---- main loop ---------------------------------------------------------------
  220. while (( $(NOW) < END_AT )); do
  221. active_windows=$(tmux list-windows -t "$SESSION" -F '#W' 2>/dev/null | grep -v '^supervisor$' || true)
  222. active_count=$(printf '%s\n' "$active_windows" | grep -c . || true)
  223. # top up to PARALLEL with not-yet-started codes
  224. while (( ${active_count:-0} < PARALLEL && idx < ${#ALL_CODES[@]} )); do
  225. code="${ALL_CODES[idx]}"
  226. idx=$((idx+1))
  227. if [[ -n "${finished[$code]:-}" ]]; then continue; fi
  228. if printf '%s\n' "$active_windows" | grep -qx "$code"; then continue; fi
  229. if (( ${restarts[$code]:-0} > MAX_RESTARTS )); then
  230. log "$code: exceeded restart budget, parking"
  231. continue
  232. fi
  233. launch "$code" && active_count=$((active_count+1)) || {
  234. if printf '%s' "${LAST_LAUNCH_ERR:-}" | grep -qi "max active"; then
  235. # platform cap — not this challenge's fault: requeue it and stop
  236. # burning through the list (a full sweep marks everything failed
  237. # and exhausts idx, after which top-up never runs again).
  238. idx=$((idx-1))
  239. log "$code: platform active cap — requeued, waiting for a free slot"
  240. break
  241. fi
  242. restarts[$code]=$((${restarts[$code]:-0}+1))
  243. log "$code: launch failed (restart ${restarts[$code]}/$MAX_RESTARTS)"
  244. }
  245. sleep 3
  246. done
  247. for win in $active_windows; do
  248. pane=$(pane_text "$win")
  249. # 1) any confirm dialog -> accept
  250. if accept_dialogs "$win" "$pane"; then
  251. last_activity[$win]=$(NOW); continue
  252. fi
  253. # 2) pane dead (claude exited)
  254. if printf '%s' "$pane" | grep -q 'DRIVE_PANE_EXITED'; then
  255. restarts[$win]=$(( ${restarts[$win]:-0} + 1 ))
  256. if (( ${restarts[$win]} > MAX_RESTARTS )); then
  257. finish_challenge "$win" "TUI exited, no restarts left"
  258. else
  259. log "$win: TUI exited — relaunching (${restarts[$win]}/$MAX_RESTARTS)"
  260. tmux kill-window -t "$SESSION:$win" 2>/dev/null
  261. launch "$win" || true
  262. fi
  263. continue
  264. fi
  265. # 3) mission text sitting unsubmitted in the input box -> re-Enter
  266. if printf '%s' "$pane" | grep -q '❯.*autonomous pentest operator'; then
  267. claim_start[$win]=$((${claim_start[$win]:-0}+1))
  268. if (( ${claim_start[$win]} > 3 )); then
  269. log "$win: mission never submits — restarting window"
  270. tmux kill-window -t "$SESSION:$win" 2>/dev/null
  271. launch "$win" || true
  272. else
  273. log "$win: mission in input box unsubmitted — re-Enter (${claim_start[$win]}/3)"
  274. tmux send-keys -t "$SESSION:$win" Enter
  275. sleep 2
  276. fi
  277. last_activity[$win]=$(NOW); continue
  278. fi
  279. # 3.5) quota/auth exhaustion in the pane -> rotate key, relaunch with resume
  280. if printf '%s' "$pane" | grep -qE "$QUOTA_RE"; then
  281. if rotate_key; then
  282. log "$win: quota/auth error in pane — key rotated, relaunching (session resumes)"
  283. restarts[$win]=${restarts[$win]:-0} # rotation is not the agent's fault: don't burn restarts
  284. tmux kill-window -t "$SESSION:$win" 2>/dev/null
  285. launch "$win" || true
  286. else
  287. log "$win: quota/auth error and no spare key — nudging later"
  288. fi
  289. last_activity[$win]=$(NOW); continue
  290. fi
  291. # 4) activity tracking + stall nudge
  292. if [[ -n "$pane" && "$pane" != "${last_pane_state[$win]:-}" ]]; then
  293. last_pane_state[$win]="$pane"; last_activity[$win]=$(NOW)
  294. else
  295. idle=$(( $(NOW) - ${last_activity[$win]:-$(NOW)} ))
  296. if (( idle > STALL_SECONDS )); then
  297. log "$win: stalled ${idle}s — nudging"
  298. send_literal "$win" "Continue autonomously. If stuck, change approach or hunt the next flag. Ensure every captured flag was submitted via sapper flag. Reply DONE only when finished."
  299. last_activity[$win]=$(NOW)
  300. fi
  301. fi
  302. # 5) DONE detection -> finish and free the slot
  303. if printf '%s' "$pane" | tail -60 | grep -q '\bDONE\b'; then
  304. sleep 8 # let trailing flag submissions land
  305. finish_challenge "$win" "agent reported DONE"
  306. continue
  307. fi
  308. # 6) per-challenge budget
  309. if (( $(NOW) - ${launch_at[$win]:-$(NOW)} > (PER_MIN * 60 + 120) )); then
  310. tmux send-keys -t "$SESSION:$win" C-c 2>/dev/null; sleep 2
  311. finish_challenge "$win" "per-challenge budget exhausted"
  312. continue
  313. fi
  314. done
  315. # all challenges finished? (re-read windows: top-up may have changed them)
  316. active_windows=$(tmux list-windows -t "$SESSION" -F '#W' 2>/dev/null | grep -v '^supervisor$' || true)
  317. if (( idx >= ${#ALL_CODES[@]} )) && [[ -z "$active_windows" ]]; then
  318. left=$(arena list 2>>"$LOG" | python3 -c '
  319. import json,sys
  320. try: items=json.load(sys.stdin)
  321. except Exception: print(999); sys.exit()
  322. print(sum(1 for c in items if not c.get("is_completed")))' 2>/dev/null || echo 999)
  323. if (( ${left:-999} == 0 )); then
  324. log "all challenges complete"; break
  325. else
  326. log "platform still shows ${left:-?} unfinished — refreshing queue"
  327. mapfile -t ALL_CODES < <(list_pending)
  328. idx=0
  329. for k in "${!finished[@]}"; do unset "finished[$k]"; done
  330. fi
  331. fi
  332. sleep 15
  333. done
  334. log "deadline reached — wrapping up"
  335. for win in $(tmux list-windows -t "$SESSION" -F '#W' 2>/dev/null | grep -v '^supervisor$'); do
  336. tmux send-keys -t "$SESSION:$win" C-c 2>/dev/null
  337. done
  338. sleep 3
  339. for win in $(tmux list-windows -t "$SESSION" -F '#W' 2>/dev/null | grep -v '^supervisor$'); do
  340. finish_challenge "$win" "deadline"
  341. done
  342. log "drive complete"

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.