Skip to content
Back to search

oh-my-agent

MarketplaceWorks with Claude CodeGitHubIncomplete

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/first-fluke/oh-my-agent with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/first-fluke/oh-my-agent'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/first-fluke--oh-my-agent/badge.svg)](/listings/first-fluke--oh-my-agent)

HTML

<a href="/listings/first-fluke--oh-my-agent"><img src="/api/listings/first-fluke--oh-my-agent/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

These steps run a script from the network that findsafeskills does not scan.

For humans

Run this from your project root.

curl -fsSL https://raw.githubusercontent.com/first-fluke/oh-my-agent/main/cli/install.sh | bash

For AI agents

Paste this into your coding agent:

Install and configure oh-my-agent in this project by following this guide:
https://raw.githubusercontent.com/first-fluke/oh-my-agent/main/docs/agent-install.md

The [agent installation guide](./docs/agent-install.md) covers non-interactive setup and verification. If you are the agent, read that guide and carry out the installation.

Then open your agent and type `/orchestrate build a TODO app with login`.

oh-my-agent (`oma`) turns Claude Code, Codex CLI, Cursor and 11 other coding agents into a team. A planner splits the work, engineers build in parallel, a reviewer checks the result, and a debugger fixes the root cause.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 37 lines
  1. {
  2. "$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
  3. "name": "oh-my-agent",
  4. "owner": {
  5. "name": "First Fluke",
  6. "url": "https://firstfluke.com/oh-my-agent"
  7. },
  8. "metadata": {
  9. "description": "Portable multi-agent harness for .agents-based skills and workflows",
  10. "version": "21.0.0"
  11. },
  12. "plugins": [
  13. {
  14. "name": "oma",
  15. "description": "All OMA skills and specialist teams",
  16. "source": "./"
  17. },
  18. {
  19. "name": "oma-frontend",
  20. "description": "Frontend implementation, interface design, review, and debugging",
  21. "category": "frontend",
  22. "source": "./.claude-plugin/domains/frontend"
  23. },
  24. {
  25. "name": "oma-security",
  26. "description": "Security scans, source review, and verified remediation",
  27. "category": "security",
  28. "source": "./.claude-plugin/domains/security"
  29. },
  30. {
  31. "name": "oma-research",
  32. "description": "Literature search, market research, document extraction, and academic writing",
  33. "category": "research",
  34. "source": "./.claude-plugin/domains/research"
  35. }
  36. ]
  37. }
README.md · README · 641 lines · 3 findings marked
  1. # oh-my-agent
  2. **Ship 400+ commits a day.**
  3. [![npm version](https://img.shields.io/npm/v/oh-my-agent?color=cb3837&logo=npm)](https://www.npmjs.com/package/oh-my-agent) [![npm downloads](https://img.shields.io/npm/dm/oh-my-agent?color=cb3837&logo=npm)](https://www.npmjs.com/package/oh-my-agent) [![GitHub stars](https://img.shields.io/github/stars/first-fluke/oh-my-agent?style=flat&logo=github)](https://github.com/first-fluke/oh-my-agent) [![License](https://img.shields.io/github/license/first-fluke/oh-my-agent)](https://github.com/first-fluke/oh-my-agent/blob/main/LICENSE) [![Last Updated](https://img.shields.io/github/last-commit/first-fluke/oh-my-agent?label=updated&logo=git)](https://github.com/first-fluke/oh-my-agent/commits/main)
  4. [한국어](./docs/README.ko.md) | [中文](./docs/README.zh.md) | [Português](./docs/README.pt.md) | [日本語](./docs/README.ja.md) | [Français](./docs/README.fr.md) | [Español](./docs/README.es.md) | [Nederlands](./docs/README.nl.md) | [Polski](./docs/README.pl.md) | [Русский](./docs/README.ru.md) | [Deutsch](./docs/README.de.md) | [Tiếng Việt](./docs/README.vi.md) | [ภาษาไทย](./docs/README.th.md)
  5. ## Install
  6. ### For humans
  7. Run this from your project root.
  8. ```bash
  9. curl -fsSL https://raw.githubusercontent.com/first-fluke/oh-my-agent/main/cli/install.sh | bash
    • Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
    • Line 16 on GitHub
  10. ```
  11. ### For AI agents
  12. Paste this into your coding agent:
  13. ```text
  14. Install and configure oh-my-agent in this project by following this guide:
  15. https://raw.githubusercontent.com/first-fluke/oh-my-agent/main/docs/agent-install.md
  16. ```
  17. The [agent installation guide](./docs/agent-install.md) covers non-interactive setup and verification. If you are the agent, read that guide and carry out the installation.
  18. Then open your agent and type `/orchestrate build a TODO app with login`.
  19. oh-my-agent (`oma`) turns Claude Code, Codex CLI, Cursor and 11 other coding agents into a team. A planner splits the work, engineers build in parallel, a reviewer checks the result, and a debugger fixes the root cause.
  20. ## Start with your task
  21. Pick a task. Load one skill into the agent you already use, or install the whole team.
  22. | Your task | Start here | First request |
  23. |---|---|---|
  24. | [Review AI-written code](./web/docs/tasks/review-ai-code.md) | `bunx skills use first-fluke/oh-my-agent@oma-qa` | "Review this diff for bugs and missing checks. Cite the affected lines." |
  25. | [Design an admin screen](./web/docs/tasks/design-admin-ui.md) | `bunx skills use first-fluke/oh-my-agent@oma-design` | "Review the hierarchy and table density of our admin page." |
  26. | [Fix a failing test](./web/docs/tasks/debug-failing-tests.md) | `bunx skills use first-fluke/oh-my-agent@oma-debug` | "Reproduce this test failure, fix its cause, and run the affected checks." |
  27. | Review system boundaries | `bunx skills use first-fluke/oh-my-agent@oma-architecture` | "Review the boundary between our API and worker." |
  28. | Run the whole team | [Install oma](#install) | `/orchestrate add login to this app` |
  29. Run the `skills use` command in your agent's terminal. It loads that one skill. For the full team with workflows, [install oma](#install). Other options are in [Choose a setup](./web/docs/getting-started/choose-setup.md).
  30. [![Oh-My-Agent: A 6+1-Layer Harness Architecture](./docs/assets/architecture/oh-my-agent-architecture.png)](./docs/assets/architecture/oh-my-agent-architecture.svg)
  31. After installing, run `oma doctor`. [Your first 5 minutes](#your-first-5-minutes) starts with a read-only task before team workflows.
  32. ## What you get
  33. **Workflows** are what you type (or just say). They call in **roles**, the specialists. **Skills** are the know-how: each role carries its own, and your agent loads the rest whenever a task needs them.
  34. | | What it gives you | Try it |
  35. |---|---|---|
  36. | **12 specialist roles** | PM, architect, backend, frontend, mobile, database and infra engineers, QA reviewer, debugger, refactorer, docs curator, researcher ([meet them](#meet-the-team)) | `/orchestrate build a TODO app with login` |
  37. | **20 workflows** | Slash commands from `/brainstorm` and `/plan` to `/review`, `/debug` and `/scm`. The four multi-agent modes also start when a plain request names them | "ultrawork: add rate limiting" |
  38. | **32 skills** | Domain know-how that loads only when a task needs it: UI, APIs, schemas, Terraform, CI, security, git, observability | loads by itself, e.g. "add a Terraform S3 bucket" |
  39. | **Files you can open** | Plans, design docs, API contracts, ADRs, bug notes, per-agent results, security evidence, HTML explainers, recaps | `ls .agents/results` |
  40. | **Research and content** | Cited search, market research, papers, slide decks, MP4 videos, images, local voice, PDF/HWP to Markdown | `/video`, `/convert` |
  41. | **One folder, 14 runtimes** | One portable `.agents/` directory, written into each runtime's own config files | `oma link cursor` |
  42. **[Install](#install)** · **[First 5 minutes](#your-first-5-minutes)** · **[The team](#meet-the-team)** · **[Which workflow?](#how-the-team-works-together)** · **[Everything in the box](#everything-in-the-box)** · **[Works with](#works-with)** · **[FAQ](#faq)** · **[Docs](https://firstfluke.com/oh-my-agent/)**
  43. ## Installation details
  44. The script installs [bun](https://bun.sh) if it is missing, installs the `oma` command globally (`bun install --global oh-my-agent`, skipped when `oma` is already on your PATH), then starts the interactive setup. The hooks (keyword detection and the Stop gate) and several workflows call the globally installed `oma` command. The hooks silently do nothing if they cannot find it. If you skip the script, install it yourself with `bun install --global oh-my-agent` (`npm install --global oh-my-agent` or `brew install oh-my-agent` also work).
  45. <details>
  46. <summary>Windows, or without the install script</summary>
  47. ```powershell
  48. # Windows (PowerShell)
  49. irm https://raw.githubusercontent.com/first-fluke/oh-my-agent/main/cli/install.ps1 | iex
  50. ```
  51. ```bash
  52. # Any OS with bun installed: the same setup, plus the global `oma` command
  53. bun install --global oh-my-agent
  54. oma
  55. ```
  56. On Windows the installer hides Antigravity, Hermes, and Kimi, and the hook wrapper is a bash script. The default Aside browser MCP installs through `bash`, so deselect it if you have no bash.
  57. </details>
  58. The setup keeps your saved response language or detects it from the system. It asks which agent runtimes to wire up, capability providers, a browser MCP, a model preset, and a skill preset. **Choose `All` for the skill references used across every workflow**: domain presets omit some of those skills. Provider availability and task verification remain separate checks.
  59. Choose a preset explicitly when you want a domain's skills:
  60. ```bash
  61. oma install --preset frontend --yes
  62. ```
  63. This selects the eight frontend-preset skills. The installer still configures the runtime, providers, and shared workflow files, and retains previously installed skills. Custom selection is available in the interactive installer. See [preset values](./web/docs/getting-started/installation.md#presets) and [workflow dependencies](./web/docs/getting-started/installation.md#workflows-that-need-skills-outside-a-preset).
  64. <details>
  65. <summary>What the installer also does by default</summary>
  66. - Installs **bun** (required). Tries **uv**, **Serena** (`uv tool install`), and **CUE** (`brew install cue` or `winget`), and continues with a warning if any of them fails.
  67. - Pre-selects the **Aside** browser MCP in the interactive setup, and installs it with `curl -fsSL https://releases.aside.com/install.sh | bash` only if you keep it selected. Deselect it in the browser MCP step, or change it later with `oma update mcp`.
    • Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
    • Line 103 on GitHub
  68. - Uses **AgentMemory** for semantic memory, which runs `bun install -g @agentmemory/agentmemory` and registers it as a login service (launchd, systemd user unit, or Windows scheduled task). Choose provider `none` or set `OMA_NO_AGENTMEMORY=1` to skip it.
  69. - Asks before linking skills into home-directory vendor folders (Antigravity, Hermes, Kimi). In `$HOME` without `--global` it asks for confirmation.
  70. - Writes some files outside the project without asking: `~/.claude.json` (marks the project as a trusted Claude Code workspace), `~/.serena/` (OMA context, answer-size floor, project registration), `~/.cursor/cli-config.json` (turns off Cursor's commit attribution), `~/.grok/config.toml` (telemetry), and `~/.gemini/antigravity-cli/` (HUD), each only for a vendor you selected.
  71. - Writes **telemetry opt-outs** into vendor settings (Claude `DISABLE_TELEMETRY=1`, which also turns off Claude Code Remote Control; Gemini, Qwen, Codex, and Grok usage stats). Set `telemetry: true` to keep vendor defaults.
  72. - Disables **Qwen Code automatic memory saving** by adding `memory.enableManagedAutoMemory: false` when the setting is absent. Explicit `true` or `false` preferences and other memory settings are preserved.
  73. - Adds unset Claude Code settings to `.claude/settings.json`, including `skipDangerousModePermissionPrompt: true`, `effortLevel: high`, `cleanupPeriodDays: 180`, and env flags that turn off auto memory, adaptive thinking, error reporting, and built-in git instructions. Values you already set are kept.
  74. - Adds a **co-author trailer**: Claude Code's commit attribution becomes `Co-Authored-By: First Fluke <our.first.fluke@gmail.com>` and PRs get a "Generated with oh-my-agent" line. Set `scm.co_author.enabled: false` to turn this off.
  75. - Installs a **commit-msg guard** in `.githooks/` that rejects unlisted `Co-authored-by` addresses, and sets `git config --local core.hooksPath .githooks` when no other hooks are active. Set `scm.co_author.enforce_hook: false` to opt out.
  76. - `oma update` updates the `oma` CLI in the background (`auto_update_cli: false` or `OMA_SKIP_VERSION_CHECK=1` to stop it) and runs `uv tool upgrade serena-agent --prerelease=allow` (`serena.auto_update: false`).
  77. - With Serena in the default `bridge` mode, the first agent session that starts a shared Serena server registers a **cleanup timer** that stops idle shared Serena servers every 5 minutes (launchd `dev.oma.serena-daemon-gc`, systemd user timer `oma-serena-daemon-gc.timer`, or Windows scheduled task `OMA Serena Daemon GC`). Set `serena.mode: stdio` to skip the shared server and its timer. `oma uninstall` does not remove the timer.
  78. - Non-interactive runs (`-y`, `OMA_YES=1`, `CI=true`) use the `All` skill preset unless `--preset` selects another, plus `model_preset: auto` and Python as the backend language. They select no browser MCP (and skip the Aside install script) unless `mcp.devtools_browsers` already lists one.
  79. </details>
  80. <details>
  81. <summary>Skill presets</summary>
  82. | Preset | Skills |
  83. |--------|--------|
  84. | **All** | **All 32 skills** (recommended) |
  85. | Fullstack | architecture, backend, brainstorm, db, debug, design, dev-workflow, frontend, mobile, pm, qa, scm, tf-infra |
  86. | Fullstack Web | Fullstack without mobile and tf-infra |
  87. | Fullstack Mobile | Fullstack without frontend and tf-infra |
  88. | Frontend | architecture, brainstorm, debug, design, frontend, pm, qa, scm |
  89. | Backend | architecture, backend, brainstorm, db, debug, dev-workflow, pm, qa, scm |
  90. | Mobile | architecture, brainstorm, debug, mobile, pm, qa, scm |
  91. | DevOps | architecture, brainstorm, debug, dev-workflow, observability, pm, qa, scm, tf-infra |
  92. | Research | academic-writing, hwp, market, pdf, scholar, scm, search, translation |
  93. | Content | academic-writing, design, image, scm, translation, voice |
  94. | Custom | Pick skills one by one |
  95. Of the named presets, only `All` includes coordination, orchestration, docs, security, refactor, explanation, recap, video, slide, and skill-creation (or pick them under `Custom`). Research and Content lack the skills behind `/plan`, `/review`, `/debug`, `/architecture`, and `/brainstorm`. See [Workflows that need skills outside a preset](./web/docs/getting-started/installation.md#workflows-that-need-skills-outside-a-preset).
  96. </details>
  97. ## Your first 5 minutes
  98. 1. **Choose your setup.** For the task below, load `oma-scm` with `bunx skills use first-fluke/oh-my-agent@oma-scm`, or use the runtime install command above. [Choose a setup](./web/docs/getting-started/choose-setup.md) explains both paths.
  99. 2. **For a runtime installation, check the install:**
  100. ```bash
  101. oma doctor
  102. ```
  103. The report shows CLI and skill-file status. Its skill counts reflect the files present, including retained skills; they can differ from the current preset's selection. If `oma` is not found, open a new terminal or run `bun install --global oh-my-agent`.
  104. 3. **Open an existing Git repository and run one read-only task.** `oma-scm` is included in every named preset; include it when using Custom selection.
  105. ```text
  106. Use oma-scm to inspect this repository's current branch and working tree. Do not edit, stage, commit, or contact remotes. Report staged, unstaged, and untracked changes, with the Git commands and exit statuses actually observed.
  107. ```
  108. 4. **Check the result.** The reported branch and changes should match the Git output. Loading instructions or completing installation does not establish that this task ran.
  109. For larger work, choose a workflow whose referenced skills are installed. Start with `/plan` when its planning skill is present, then continue with `/work` or `/orchestrate` when their dependencies are available.
  110. ## Meet the team
  111. Your own agent leads the team. In `/work`, `/orchestrate` and `/ultrawork` it hands each job to the right specialist, which runs as its own subagent. `/plan`, `/architecture` and `/docs` run in your current session with that role's skill, and `/review` and `/debug` call in their specialist for large scopes. When a role is on another vendor's CLI, `oma agent spawn` runs it there.
  112. | Your specialist | Carries | Call them with | Leaves behind |
  113. |---|---|---|---|
  114. | **Product manager** (`pm-planner`) | oma-pm | `/plan` | A plan JSON and a tracker in `docs/plans/work/` for Medium and Complex work (Simple requests get a direct recommendation) |
  115. | **Architect** (`architecture-reviewer`) | oma-architecture | `/architecture` | An ADR or recommendation in `.agents/results/architecture/` |
  116. | **Backend, frontend, mobile, database and infra engineers** | oma-backend, oma-frontend + oma-design, oma-mobile, oma-db, oma-tf-infra | `/work`, `/orchestrate` | Code, tests, and a `result-<agent>` file per task |
  117. | **QA reviewer** (`qa-reviewer`) | oma-qa | `/review` | Findings ranked CRITICAL to LOW, each with file:line and fix code |
  118. | **Debugger** (`debug-investigator`) | oma-debug | `/debug` | A minimal fix, a regression test, and a bug note in `.agents/results/bugs/` when one is warranted |
  119. <details>
  120. <summary>The other three roles, and the skills without a role</summary>
  121. | Your specialist | Carries | Call them with | Leaves behind |
  122. |---|---|---|---|
  123. | **Refactorer** (`refactor-engineer`) | oma-refactor | `/ultrawork`'s REFINE phase | Behavior-preserving `refactor:` commits and a `result-refactor-<session>.md` report |
  124. | **Docs curator** (`docs-curator`) | oma-docs | `/docs`, or "verify docs" | `docs/generated/doc-refs.json` and the doc patches you approve |
  125. | **Researcher** (`research-explorer`) | oma-search, oma-market, oma-scholar | a research request | Cited sources with their provenance; market briefs in `.agents/results/market/` |
  126. The infra engineer (`tf-infra-engineer`) is told to ask before any destructive or production `apply` or `destroy`. The remaining skills (brainstorm, security, scm, dev-workflow, observability, explanation, recap, translation, coordination, orchestration, skill-creation, content and media) have no dedicated role. Your agent loads them itself, and for most of them the skill injector suggests the right one from your prompt.
  127. </details>
  128. ## How the team works together
  129. ```text
  130. THINK /brainstorm ......................... docs/decisions/<NNN>-<feature>.md
  131. PLAN /plan · /architecture ............... plan JSON, API contracts, ADR
  132. DESIGN /design ............................. DESIGN.md + tokens
  133. BUILD /work · /orchestrate · /ultrawork ... code + result-<agent> files, engineers in parallel
  134. CHECK /review · /security · /debug ........ findings, security evidence, regression test
  135. SHIP /scm · /docs ........................ Conventional Commits, doc fixes
  136. ```
  137. Each step hands a file to the next. `/brainstorm` saves a design that `/plan` reads, and `/plan` writes a plan in which every task has an agent, dependencies, acceptance criteria and a check command. `/orchestrate` runs that plan (it loads the most recent `plan-*.json`, or writes one). `/work` and `/ultrawork` start by having the PM write their own plan. Skip any stage you don't need; every stage also works on its own.
  138. **Which workflow should run the team?**
  139. | You want | Run | How it ends |
  140. |---|---|---|
  141. | Just a plan | `/plan` | A plan file for multi-area work, or a direct recommendation for small work |
  142. | A PM-led build you follow step by step | `/work` | PM, then engineers tier by tier, then QA. CRITICAL and HIGH findings go into a fix loop (up to 5 cycles) |
  143. | A hands-off parallel build | `/orchestrate` | Only tasks whose checks pass count as done; each task gets up to 3 attempts, then is marked partial or failed |
  144. | Five-phase workflow | `/ultrawork` | Five gated phases (PLAN, IMPL, VERIFY, REFINE, SHIP); every review runs in a fresh reviewer session |
  145. | "Don't stop until it's really done" | `/ralph` | Repeats `/ultrawork` with a separately spawned, fresh-context judge until every criterion passes; stops with a partial report when criteria are blocked or after 5 iterations |
  146. | The team on a timer | `oma schedule create qa-reviewer "review open PRs" --every 2h` | One Markdown log per run in `~/.oma/schedule/runs/<id>/` |
  147. A slash command starts a workflow. A plain request does not, but on runtimes with a prompt hook it still gets a skill hint: keyword banks in up to 11 languages match the prompt and point your agent at the matching skills ("run a security audit" brings in `oma-qa`). The hint is advisory, and keywords inside code blocks are ignored.
  148. `/orchestrate`, `/work`, `/ultrawork` and `/ralph` are **persistent** when you invoke them by name, as a slash command or in plain text ("ultrawork: add rate limiting"). The agent is then sent back to work when it tries to stop early (up to 5 times or 2 hours) until you say **"workflow done"**. These four are the only workflows a plain sentence can start, and only by name.
  149. **Measured, with caveats.** In the [benchmark](./benchmarks/README.md) (one prompt, Claude Code with five different add-on setups including plain, `claude-opus-4-6`, one build each), oma scored 80.6 against 70.7 for plain Claude Code, at 1.7x the cost ($4.04 vs $2.37) and 1.8x the time.
  150. <details>
  151. <summary>All 20 workflows</summary>
  152. | Command | What it does |
  153. |---------|--------------|
  154. | `/architecture` | Weighs design tradeoffs and produces a recommendation, review, or ADR |
  155. | `/brainstorm` | Explores ideas and constraints and produces an approved design before planning |
  156. | `/convert` | Converts PDF/HWP to Markdown, or transcodes images, video, and audio with ffmpeg |
  157. | `/debug` | Reproduces a bug, fixes the root cause, writes a regression test, and scans for similar patterns |
  158. | `/deepinit` | Maps an existing codebase into AGENTS.md, ARCHITECTURE.md, and a docs/ knowledge base |
  159. | `/design` | Scopes a design task and delivers findings, DESIGN.md, or tokens through oma-design |
  160. | `/docs` | Finds broken doc references and proposes patches for docs a diff touched |
  161. | `/explain` | Turns a diff, PR, branch, or topic into a self-contained interactive HTML explainer |
  162. | `/orchestrate` | Runs agents in parallel, coordinates them through file state, and verifies the result (persistent) |
  163. | `/plan` | Breaks a feature into prioritized tasks with API contracts |
  164. | `/ralph` | Repeats `/ultrawork` with a separately spawned judge, up to 5 iterations (persistent) |
  165. | `/recap` | Summarizes your AI conversation history for a day or period |
  166. | `/review` | Reviews security (OWASP Top 10), performance, accessibility (WCAG 2.2 AA), and code quality |
  167. | `/schedule` | Registers an agent job that runs on a recurring interval |
  168. | `/scm` | Handles branches, merges, conflicts, worktrees, and Conventional Commits |
  169. | `/security` | Scans source, agent skills, and MCP components, validates findings, and sets up CI gates |
  170. | `/stack-set` | Detects your tech stack and generates stack-specific references for domain skills |
  171. | `/ultrawork` | Five phases (PLAN, IMPL, VERIFY, REFINE, SHIP), each review in a fresh reviewer session (persistent) |
  172. | `/video` | Turns a brief into a finished MP4 through a key-optional pipeline |
  173. | `/work` | Coordinates several agents on a multi-domain project with PM planning and QA review (persistent) |
  174. `oma verify triggers` evaluates the detector against a labeled English and Korean corpus of 212 prompts. CI fails above 10% false fires or 20% missed fires. Last measured: **0% missed-fire, 0% false-fire**. Full reference: [Workflows](https://first-fluke.github.io/oh-my-agent/docs/core-concepts/workflows/).
  175. </details>
  176. ## Everything in the box
  177. <details>
  178. <summary><strong>Plan and design</strong>: <code>/brainstorm</code>, <code>/plan</code>, <code>/architecture</code>, <code>/design</code></summary>
  179. | Command | What it does | Writes |
  180. |---|---|---|
  181. | `/brainstorm` | Asks one clarifying question at a time and proposes 2-3 approaches that work differently, then hands off to `/plan`. Writes no code | `docs/decisions/<NNN>-<feature>.md` |
  182. | `/plan` | Sizes the work as Simple, Medium or Complex; for Medium and Complex work, splits it into tasks, each with an agent, dependencies, acceptance criteria and a check command, and defines API contracts when the work crosses a boundary | `.agents/results/plan-<sid>.json`, `docs/plans/work/<NNN>-<name>.md`, contracts in `.agents/results/api-contracts/` when needed |
  183. | `/architecture` | Reads earlier decisions first, compares at least 2 options, and marks replaced ADRs as superseded | An ADR or recommendation in `.agents/results/architecture/` |
  184. | `/design` | Establishes scope and context, selects the design guidance for the task, then delivers audit findings, `DESIGN.md`, or tokens mapped onto the project's existing token system | Findings, `DESIGN.md`, design tokens |
  185. </details>
  186. <details>
  187. <summary><strong>Build</strong>: 7 domain skills, plus <code>/stack-set</code> to fit them to your stack</summary>
  188. | Skill | Builds |
  189. |---|---|
  190. | oma-frontend | React/Next.js or Angular UI that follows your tokens, i18n and accessibility rules |
  191. | oma-backend | APIs, auth and data access, layered router → service → repository |
  192. | oma-mobile | Flutter, React Native or Swift apps |
  193. | oma-db | Schemas, migrations, query tuning, vector/RAG retrieval plans |
  194. | oma-tf-infra | Terraform for any cloud, with drift, destructive-change and cost risks called out |
  195. | oma-dev-workflow | mise tasks, lint/format/typecheck, commitlint, git hooks, CI/CD and releases |
  196. | oma-observability | OpenTelemetry, SLOs, alerting and incident forensics |
  197. `/stack-set` reads your manifests and writes stack-specific references for the backend, frontend and mobile skills. In a monorepo, frontend, backend and mobile engineers are pointed at their workspace (pnpm, npm/yarn/bun workspaces, lerna, nx, turbo, `mise.toml`).
  198. </details>
  199. <details>
  200. <summary><strong>Review, fix and ship</strong>: <code>/review</code>, <code>/security</code>, <code>/debug</code>, refactor, <code>/scm</code></summary>
  201. | Command or skill | What it does | Writes |
  202. |---|---|---|
  203. | `/review` | A dependency audit, then OWASP Top 10, performance, WCAG 2.2 AA and code quality. Every finding gets file:line and fix code | The report in chat plus a memory record; no report file |
  204. | `/security` | Scans your source, agent skills and MCP servers (Deepsec, Cisco Skill and MCP Scanner) and validates the findings. Penetration tests run only on explicit request, against a runtime platform you supply and have reviewed. The workflow requires an engine that failed to run to be reported as failed, not as clean | `.agents/results/security/<run-id>/` |
  205. | `/debug` | Reproduces the failure, fixes the root cause with a minimal change, adds a regression test, and scans for the same pattern elsewhere | The fix and test; a bug note in `.agents/results/bugs/` when one is warranted |
  206. | oma-refactor | Changes structure without changing behavior: characterization tests first, then refactor-only commits | `.agents/results/refactor/` |
  207. | `/scm` | Conventional Commits split by logical change; branches, merges, rebases and worktrees | Commits and branches |
  208. </details>
  209. <details>
  210. <summary><strong>Docs and knowledge</strong>: <code>/deepinit</code>, <code>/docs</code>, <code>/explain</code>, <code>/recap</code>, translation</summary>
  211. | Command or skill | What it does | Writes |
  212. |---|---|---|
  213. | `/deepinit` | Turns an existing repo into an agent-ready knowledge base | `AGENTS.md`, `ARCHITECTURE.md`, `docs/` |
  214. | `/docs` | Finds broken references, patches the docs a diff touched (only the patches you approve), and finds stale translations | `docs/generated/doc-refs.json`, patched docs |
  215. | `/explain` | Turns a PR, branch or staged change into an HTML walkthrough with a 5-question quiz, or a topic into a one-page visual explainer | `.agents/results/explain/<date>-<slug>.html` |
  216. | `/recap` | Summarizes your AI conversations across tools for a day or a period | `.agents/results/recap/<date>.md` |
  217. | oma-translation | Translation that reads natively; placeholders, code and locale keys stay unchanged | Translated text |
  218. </details>
  219. <details>
  220. <summary><strong>Research</strong>: search, market, papers, academic writing</summary>
  221. | Skill | What it does | Writes |
  222. |---|---|---|
  223. | oma-search | Docs, web and code search, noting what was read versus only excerpted | Cited results |
  224. | oma-market | Pain points, trends and competitors from community signals, framed with SWOT, Porter's 5 Forces and PESTEL | `.agents/results/market/<topic>-<date>.md` |
  225. | oma-scholar | Paper search that reads a compact claims file (Knows format, about 700 tokens) instead of the full PDF (about 10K tokens) | `<paper>.knows.yaml` |
  226. | oma-academic-writing | Drafts and audits academic prose against a rubric | Your file |
  227. </details>
  228. <details>
  229. <summary><strong>Content and media</strong>: <code>/video</code>, <code>/convert</code>, slides, images, voice</summary>
  230. | Command or skill | What it makes | Writes |
  231. |---|---|---|
  232. | `/video` | Shorts, explainers or demos as a finished MP4; keyed providers have key-free fallbacks | `.agents/results/videos/<run>/` with a manifest |
  233. | oma-slide | 1920×1080 HTML decks, exported to PDF, PNG or PPTX | `.agents/results/slides/<session>/` |
  234. | oma-image | Images through the vendor CLIs you're signed into, compared side by side | `.agents/results/images/<run>/` |
  235. | oma-voice | Local text-to-speech and transcription through the Voicebox app | `.agents/results/voice/<run>/` |
  236. | `/convert` | PDF or HWP to Markdown; image, video and audio conversion through ffmpeg | Next to the input file |
  237. These pipelines report a missing source or vendor key instead of silently returning a thinner result.
  238. </details>
  239. <details>
  240. <summary><strong>Hooks that run without being asked</strong></summary>
  241. | Hook | What it does | Off switch / limit |
  242. |---|---|---|
  243. | Keyword start | Starts `/orchestrate`, `/work`, `/ultrawork` or `/ralph` when a plain request names it | Questions, pasted text and code blocks are skipped |
  244. | Skill injector | Points your agent at the right skills for the prompt | Paused while a persistent workflow is active |
  245. | Stop gate | Sends a persistent workflow back to work; with `oma goal set --gate test`, the stop is allowed when that script exits 0 within 25 s | Say "workflow done"; it also releases after 5 blocks or 2 hours |
  246. | Refactor guard | At stop, flags edited files over 500 lines for `refactor-engineer` | Opt-in: `refactor_guard.enabled` |
  247. | Secret guard | Blocks `git add` of `*.env`, `*.pem`, `id_rsa*`, `*.tfstate` and similar files | Prefix that one command with `OMA_SCM_ALLOW_SECRETS=1` |
  248. | Code-search guard | Blocks native recursive search in the project so the agent uses Serena or Gortex | `providers.code_intelligence_guard: off` |
  249. | Test filter | Keeps only failures and summaries from test output, so fewer tokens are spent | `OMA_TEST_FILTER=0` |
  250. | Session memory | When you switch or resume a session (or after compaction), re-injects the active session's recent events plus AgentMemory recall | Memory provider `none` or `OMA_NO_AGENTMEMORY=1` turns off the recall; the local event snapshot stays |
  251. `oma hook probe` checks that oma's prompt hooks accept each runtime's hook input (it does not launch the runtime). Which hooks each runtime wires is in [Host differences](#host-differences).
  252. </details>
  253. <details>
  254. <summary><strong>Parallel agents, sessions and dashboards</strong></summary>
  255. - `oma agent spawn <role> "<task>" <sid>` runs a role on Antigravity, Claude Code, Codex, Cursor, Grok, Kimi Code, Kiro, OpenCode, pi or Qwen Code, with `--isolation worktree` and `--fallback-vendors`. `oma agent parallel` runs several at once. These 10 dispatch targets are a subset of the 14 installer integrations.
  256. - Add `--read-only` to require a vendor restriction before launch. Codex uses its read-only sandbox, Claude uses plan mode, and Pi excludes `edit`, `write`, `bash`, and `powershell`. Other CLIs require an explicit `read_only_flag`; missing, empty, or known conflicting flags stop dispatch. Custom flags declare support in your CLI or wrapper. Pi's tool exclusions do not restrict arbitrary extension or MCP tools; OMA does not add an operating-system sandbox.
  257. - `oma state list --all-projects` and `oma state get <sid>`: sessions and every recorded event.
  258. - `oma dashboard terminal` / `oma dashboard web`: live agent status (the web dashboard listens on `127.0.0.1` only). [Dashboard guide](./web/docs/guide/dashboard-monitoring.md).
  259. - `oma stats get` (productivity metrics), `oma retro` (a git retrospective), `oma recap --window 7d` (what you did across AI tools).
  260. - `oma visualize --affected <paths>`: which skills, workflows, agents and checks depend on the `.agents/` files you changed.
  261. </details>
  262. <details>
  263. <summary><strong>Code intelligence, memory and MCP</strong></summary>
  264. | Slot | Default | Alternatives |
  265. |---|---|---|
  266. | Docs | Context7 | — |
  267. | Web search | Runtime native search | Brave Search |
  268. | Code intelligence | Serena, one shared server per project | Gortex (experimental) |
  269. | Semantic memory | AgentMemory | Honcho (experimental), none |
  270. | Browser MCP | Aside | Chrome DevTools, Firefox DevTools |
  271. Serena supplies LSP-based code navigation and diagnostics, including native-edit feedback where supported. Provider selection is independent of the agent runtime; capabilities vary by provider and host.
  272. Agents coordinate through plain files in `.agents/state/`, with no outside service. Details: [Capability Providers](./docs/capability-providers.md).
  273. </details>
  274. <details>
  275. <summary><strong>Getting better over time</strong></summary>
  276. - `oma skill eval` measures a skill's lift over a baseline on held-out tasks. `oma skill optimize` proposes edits that raise it, and `--apply` keeps them ([skill-eval guide](./web/docs/guide/skill-eval.md)).
  277. - `oma harness evolution enable --max-dispatches <n>` schedules a nightly cycle that turns failed runs into regression fixtures and applies skill improvements that pass evaluation as project overlays. Add `--mode propose` to only record them.
  278. - `oma emit` exports the `.agents/` source as an Agent Skills package, an Agent Plugins package, a Claude Code plugin marketplace, or `AGENTS.md`.
  279. </details>
  280. <details>
  281. <summary><strong>Config and safety rails</strong></summary>
  282. - One user-owned `.agents/oma-config.yaml` survives updates (only `oma update --force` resets it); private `.local.yaml` or `.local.cue` overlays can sit on top of it. `language` sets the response language.
  283. - 13 project rules (backend, commit, database, debug, design, and more) load as native rules in Claude Code and Cursor, and are indexed in the managed `AGENTS.md` block written for Codex, Qwen Code and Pi.
  284. - Updates keep config and migration backups in `.agents/backup/`; shipped skills and workflows are replaced in place, so keep local changes outside them. `oma uninstall --dry-run` shows what would be removed. `first-fluke/oma-update-action@v1` runs `oma update` in CI and opens a PR.
  285. </details>
  286. <details>
  287. <summary><strong>All 32 skills</strong></summary>
  288. | Skill | What it does |
  289. |---|---|
  290. | **oma-academic-writing** | Drafts, revises, and audits academic prose. |
  291. | **oma-architecture** | Weighs architecture tradeoffs and draws module boundaries, with ADR/ATAM/CBAM analysis. |
  292. | **oma-backend** | Builds APIs, authentication, and data access in Python, Node.js, or Rust. |
  293. | **oma-brainstorm** | Explores ideas with you before you commit to building. |
  294. | **oma-db** | Designs schemas, migrations, indexes, and vector stores. |
  295. | **oma-debug** | Finds the root cause, fixes the bug, and writes a regression test. |
  296. | **oma-design** | Builds design systems with tokens, accessibility, and responsive layouts. |
  297. | **oma-dev-workflow** | Automates CI/CD, releases, and monorepo tasks. |
  298. | **oma-docs** | Checks docs for broken references, syncs them with code changes, and flags translation drift. |
  299. | **oma-explanation** | Turns a diff, PR, branch, or topic into a self-contained HTML explainer, with a quiz for code changes. |
  300. | **oma-frontend** | Builds UI in React/Next.js or Angular. |
  301. | **oma-hwp** | Converts HWP, HWPX, and HWPML files to Markdown. |
  302. | **oma-image** | Generates images and reference-guided variations, and can compare AI vendors side by side. |
  303. | **oma-market** | Researches your market from community signals and frames it with SWOT, 5F, and PESTEL. |
  304. | **oma-mobile** | Builds mobile apps with Flutter, React Native, or native Swift. |
  305. | **oma-observability** | Routes observability work across metrics, logs, traces, SLOs, and incident forensics. |
  306. | **oma-orchestration** | Runs multiple agents in parallel from the CLI, automatically or one dispatch at a time. |
  307. | **oma-pdf** | Converts PDF files to Markdown. |
  308. | **oma-pm** | Plans tasks, breaks down requirements, and defines API contracts. |
  309. | **oma-qa** | Reviews changes for correctness, OWASP security, performance, and accessibility. |
  310. | **oma-recap** | Recaps your conversation history into themed work summaries. |
  311. | **oma-refactor** | Refactors without changing behavior, using hotspot targeting, characterization tests, and refactor-only commits. |
  312. | **oma-scholar** | Searches academic literature and builds Knows paper sidecars for claims, evidence, and peer review. |
  313. | **oma-scm** | Manages branches, merges, worktrees, and Conventional Commits. |
  314. | **oma-search** | Routes each query to the best source and reports where each result came from and whether it was actually read. |
  315. | **oma-security** | Scans source, agent skills, and MCP components and validates findings; penetration tests run only on a user-supplied, reviewed platform. |
  316. | **oma-skill-creation** | Writes and audits new OMA skills. |
  317. | **oma-slide** | Generates HTML presentation decks and exports them to PDF/PNG/PPTX. |
  318. | **oma-tf-infra** | Provisions multi-cloud infrastructure with Terraform. |
  319. | **oma-translation** | Translates so the result reads like a native speaker wrote it. |
  320. | **oma-video** | Generates short-form, explainer, and demo videos through a key-optional HyperFrames pipeline. |
  321. | **oma-voice** | Generates voiceovers and transcribes audio on-device. |
  322. </details>
  323. ## Works with
  324. 14 installer targets: Claude Code, Codex CLI, Cursor, Antigravity, Qwen Code, Grok, Kiro, Kimi, OpenCode, Pi, GitHub Copilot, Hermes, CommandCode, and ZCode. Wire a runtime you skipped with `oma link <vendor>`. Runtimes differ in which hooks they offer; the full matrix is in [Supported Agents](./docs/SUPPORTED_AGENTS.md).
  325. - **Orca IDE:** [OMA for Orca](./integrations/orca/README.md) adds a sidebar panel and command-palette actions (Orca 1.4.197+ with experimental plugins).
  326. - **DeepSeek Harness:** [OMA for DeepSeek Harness](./integrations/dsh/README.md) connects OMA checks through a Cordis plugin. It needs `oma` on your `PATH`.
  327. <details>
  328. <summary>Runtime logos (installer targets plus Amp and DeepSeek Harness)</summary>
  329. native = oma writes the runtime's own config files; adapter = oma also adds hook or agent shims. Amp reads `.agents/skills/` by itself and is not an installer target.
  330. <table>
  331. <colgroup>
  332. <col span="6" style="width:16.67%" />
  333. </colgroup>
  334. <tr>
  335. <td align="center">
  336. <a href="https://claude.com/product/claude-code"><img src="https://github.com/anthropics.png?size=120" alt="Claude Code" width="48" height="48" /></a><br/>
  337. <strong>Claude Code</strong><br/>
  338. <sub>native + adapter</sub>
  339. </td>
  340. <td align="center">
  341. <a href="https://github.com/openai/codex"><img src="https://github.com/openai.png?size=120" alt="Codex CLI" width="48" height="48" /></a><br/>
  342. <strong>Codex CLI</strong><br/>
  343. <sub>native + adapter</sub>
  344. </td>
  345. <td align="center">
  346. <a href="https://antigravity.google"><img src="./docs/assets/agents/antigravity.png" alt="Antigravity" width="48" height="48" /></a><br/>
  347. <strong>Antigravity</strong><br/>
  348. <sub>native SSOT</sub>
  349. </td>
  350. <td align="center">
  351. <a href="https://cursor.com"><img src="https://github.com/cursor.png?size=120" alt="Cursor" width="48" height="48" /></a><br/>
  352. <strong>Cursor</strong><br/>
  353. <sub>native + adapter</sub>
  354. </td>
  355. <td align="center">
  356. <a href="https://github.com/QwenLM/qwen-code"><img src="https://github.com/QwenLM.png?size=120" alt="Qwen Code" width="48" height="48" /></a><br/>
  357. <strong>Qwen Code</strong><br/>
  358. <sub>native dispatch</sub>
  359. </td>
  360. <td align="center">
  361. <a href="https://github.com/deepseek-ai/deepseek-harness"><img src="https://github.com/deepseek-ai.png?size=120" alt="DeepSeek Harness" width="48" height="48" /></a><br/>
  362. <strong>DeepSeek Harness</strong><br/>
  363. <sub>Cordis plugin</sub>
  364. </td>
  365. </tr>
  366. <tr>
  367. <td align="center">
  368. <a href="https://pi.dev/"><img src="./docs/assets/agents/pi.svg" alt="Pi" width="48" height="48" /></a><br/>
  369. <strong>Pi</strong><br/>
  370. <sub>native-compatible</sub>
  371. </td>
  372. <td align="center">
  373. <a href="https://github.com/anomalyco/opencode"><img src="./docs/assets/agents/opencode.png" alt="OpenCode" width="48" height="48" /></a><br/>
  374. <strong>OpenCode</strong><br/>
  375. <sub>native-compatible</sub>
  376. </td>
  377. <td align="center">
  378. <a href="https://ampcode.com"><img src="./docs/assets/agents/amp.png" alt="Amp" width="48" height="48" /></a><br/>
  379. <strong>Amp</strong><br/>
  380. <sub>reads .agents/skills</sub>
  381. </td>
  382. <td align="center">
  383. <a href="https://github.com/features/copilot"><img src="https://github.com/github.png?size=120" alt="GitHub Copilot" width="48" height="48" /></a><br/>
  384. <strong>GitHub Copilot</strong><br/>
  385. <sub>symlinked skills</sub>
  386. </td>
  387. <td align="center">
  388. <a href="https://grok.x.ai"><img src="./docs/assets/agents/grok.png" alt="Grok Build" width="48" height="48" /></a><br/>
  389. <strong>Grok Build</strong><br/>
  390. <sub>native hooks</sub>
  391. </td>
  392. <td align="center">
  393. <a href="https://kiro.dev"><img src="./docs/assets/agents/kiro.png" alt="Kiro CLI" width="48" height="48" /></a><br/>
  394. <strong>Kiro CLI</strong><br/>
  395. <sub>native hooks + agents</sub>
  396. </td>
  397. </tr>
  398. </table>
  399. <p align="center"><sub><a href="./docs/SUPPORTED_AGENTS.md">& more</a></sub></p>
  400. </details>
  401. <a id="host-differences"></a>
  402. <details>
  403. <summary>Host differences: which runtimes get keyword detection and the Stop gate</summary>
  404. | Runtime | Keyword detection | Stop gate |
  405. |---|---|---|
  406. | Claude Code, Codex CLI, Qwen Code, Grok, Kimi | yes | yes |
  407. | Cursor | yes | yes (re-enters as a follow-up message) |
  408. | Antigravity | yes | yes (runs handlers through bun, not the `oma` wrapper) |
  409. | OpenCode, Pi | yes | best-effort (Pi needs a build with `agent_settled`) |
  410. | Kiro | yes | no: Kiro ignores Stop hook output |
  411. | CommandCode | no prompt hook | yes |
  412. | GitHub Copilot, Hermes, ZCode | no hooks | no hooks |
  413. </details>
  414. <a id="per-agent-models"></a>
  415. <details>
  416. <summary>Per-agent models</summary>
  417. `model_preset` in `.agents/oma-config.yaml` chooses which model each agent role uses. The default, `auto`, follows the running runtime's own agent and model settings.
  418. ```yaml
  419. language: en
  420. model_preset: auto # auto | free | antigravity | claude | codex | cursor | kiro | mixed | qwen | <custom_presets key>
  421. # Optional per-agent overrides; they win over the preset
  422. agents:
  423. backend: { model: openai/gpt-6.1-terra, effort: high }
  424. ```
  425. - `mixed` uses Claude for orchestrator, architecture, QA, PM, and docs, and Codex `gpt-6.1-terra` for implementation roles, so it needs both Claude Code and Codex CLI.
  426. - `free` routes through a FreeLLMAPI gateway and needs `FREELLM_API_KEY` plus a running server.
  427. - `kiro` is not in the installer menu; set it in the config file.
  428. - `oma doctor --profile` prints the resolved model for each role.
  429. - Full guide: [Per-agent models](https://first-fluke.github.io/oh-my-agent/docs/guide/per-agent-models/)
  430. </details>
  431. ## FAQ
  432. <details>
  433. <summary><strong>Do I need every skill?</strong></summary>
  434. No. See [Skill presets](#install); `All` is the safe default.
  435. </details>
  436. <details>
  437. <summary><strong>How do I know it is working?</strong></summary>
  438. `oma doctor` shows installed skills and warns when hooks can't find `oma`. During a persistent workflow, `oma state list` shows the active session and `oma state get <sid>` shows its gate events.
  439. </details>
  440. <details>
  441. <summary><strong>Does it cost more tokens?</strong></summary>
  442. Usually, yes. Skills load only when a task needs them, but multi-agent workflows and fresh-context reviews spend more. In the [one-prompt benchmark](./benchmarks/README.md), oma cost 1.7x plain Claude Code.
  443. </details>
  444. <details>
  445. <summary><strong>The Stop hook keeps blocking. How do I get out?</strong></summary>
  446. Say "workflow done" (or "workflow complete" / "workflow finished", or the equivalent in the other supported languages), or delete `.agents/state/<workflow>-state-<sid>.json`. If you do neither, it releases on its own after 5 blocks or 2 hours.
  447. </details>
  448. <details>
  449. <summary><strong>How do I update or uninstall?</strong></summary>
  450. `oma update` refreshes skills, workflows, and hooks and keeps your config (`--with-new-skills` adds skills new in the release; `oma update mcp` re-chooses the browser MCP). `oma uninstall --dry-run` shows what would be removed; `oma uninstall` removes it and keeps `oma-config.yaml`, `mcp.json`, and skills you wrote. More fixes: [Troubleshooting](https://first-fluke.github.io/oh-my-agent/docs/guide/troubleshooting/).
  451. </details>
  452. <details>
  453. <summary><strong>What does oh-my-agent leave to the model?</strong></summary>
  454. Planning and choosing the next step stay with your agent's model; oma has no fixed workflow graph. What oma enforces outside the model: tool guardrails (secret and search guards), budgets, retry and stop limits, an event log, and, when you set a gate, a stop that waits for your own checks to pass.
  455. </details>
  456. ## Learn more
  457. - **[Docs site](https://first-fluke.github.io/oh-my-agent/)**: guides, tutorials, and CLI reference. Start with [Quick Start](https://first-fluke.github.io/oh-my-agent/docs/getting-started/quick-start/) and [Installation](https://first-fluke.github.io/oh-my-agent/docs/getting-started/installation/).
  458. - **[Why oh-my-agent](https://first-fluke.github.io/oh-my-agent/docs/getting-started/why-oh-my-agent/)**: when to choose it and the trade-offs
  459. - **[Benchmark Report](./benchmarks/README.md)**: how it compares with superpowers, ECC, and oh-my-claudecode on one prompt, with method, scores, screenshots, and caveats
  460. - **[Supported Agents](./docs/SUPPORTED_AGENTS.md)**: support matrix across runtimes
  461. - **[Detailed Documentation](./docs/AGENTS_SPEC.md)**: technical spec and architecture
  462. - **[Capability Providers](./docs/capability-providers.md)**: experimental Gortex and Honcho configuration, routing, and limits
  463. - **[Skill Compatibility Matrix](./docs/skills-matrix.md)**: native skill content, references, and response contracts in Claude and Codex
  464. > Star the project on GitHub:
  465. >
  466. > ```bash
  467. > gh api --method PUT /user/starred/first-fluke/oh-my-agent
  468. > ```
  469. >
  470. > Starter template: [fullstack-starter](https://github.com/first-fluke/fullstack-starter)
  471. ## Star History
  472. [![Star History Chart](https://star-history.dera.page/svg?repos=first-fluke/oh-my-agent&type=date&legend=bottom-right)](https://star-history.dera.page/#first-fluke/oh-my-agent&type=date&legend=bottom-right)
  473. ## References
  474. <details>
  475. <summary>Research this builds on (7 papers)</summary>
  476. - Li, X., Liu, Y., Chen, W., You, B., Di, Z., He, Y., Zheng, S., Choe, K. W., Sun, J., Wang, S., Tao, C., Li, B., Zhao, X., Geng, H., Wu, X., Zhou, J., Chen, X., Xing, H., Li, Y., … Song, D. (2026). *SkillsBench: Benchmarking how well agent skills work across diverse tasks* (Version 4) [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2602.12670
  477. - Yu, G., & Wang, X. (2026). *Knows: Agent-native structured research representations* (Version 1) [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2604.17309
  478. - Liang, Q., Wang, H., Liang, Z., & Liu, Y. (2026). *From skill text to skill structure: The scheduling-structural-logical representation for agent skills* (Version 4) [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2604.24026
  479. - Chen, C., Yu, Q., Gu, Y., Huang, Z., Li, H., Liu, H., Liu, S., Liu, J., Peng, D., Wang, J., Yan, Z., Meng, F., Qin, E., Che, C., & Hu, M. (2026). *The scaling laws of skills in LLM agent systems* (Version 1) [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2605.16508
  480. - Tang, L., Rashtchian, C., Ferng, C.-S., Tomkins, A., Juan, D.-C., & Vu, T. (2026). *WikiSkill: Compiling agent experience into persistent knowledge for skill evolution* [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2608.27454
  481. - Huang, Z., Xu, J., Yang, Y., Gong, Z., Yang, Q., Tian, M., Wang, X., Lv, C., Gao, X., Dai, Q., Liu, B., Qiu, K., Yang, X., Chen, D., Zheng, X., & Luo, C. (2026). *From raw experience to skill consumption: A systematic study of model-generated agent skills* [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2605.23899
  482. - Hong, D. B., Imani, A., & Ahmed, I. (2026). *From anatomy to smells: An empirical study of SKILL.md in agent skills* (Version 2) [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2607.01456
  483. </details>
  484. ## License
  485. MIT
benchmarks/external/workbuddy/run.py · Bundled script · 331 lines · 1 finding marked
  1. #!/usr/bin/env python3
  2. """Run one WorkBuddy Code task with OMA/Codex and its unmodified native verifier.
  3. This is a local smoke run, not the canonical WorkBuddy Harbor job launcher.
  4. Only downloaded binaries, apt packages, and binary Python wheels are installed.
  5. """
  6. from __future__ import annotations
  7. import argparse
  8. import base64
  9. import hashlib
  10. import io
  11. import json
  12. import os
  13. from pathlib import Path
  14. import platform
  15. import shutil
  16. import subprocess
  17. import sys
  18. import tarfile
  19. import time
  20. import urllib.request
  21. import zipfile
  22. UPSTREAM = "https://github.com/Tencent/workbuddy-bench.git"
  23. REVISION = "625b2233093ae4f23e76be28c1f341d41cc70373"
  24. TASK = "bug_fix-easy-a_crash_in_local"
  25. DATASET = "wb-bench-code-v1.0"
  26. DATA_SHA256 = "f60394e44341f45f66af17ba69efc51ceadc61f17c08aa89e4af35b51f6f34e1"
  27. OMA_VERSION = "20.10.2"
  28. CODEX_VERSION = "0.163.0"
  29. NODE_VERSION = "24.11.1"
  30. CUE_VERSION = "0.17.1"
  31. BUN_VERSION = "1.3.14"
  32. BASE_IMAGE = "python:3.12-slim"
  33. GENERAL_IMAGE_ID = "sha256:ce9a404c2c0138e747a43e6ea022d2f7e670ed868df35d627a663ba7fb940ea9"
  34. GENERAL_IMAGE_DIGEST = "python@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1"
  35. GENERAL_PLATFORM = "linux/arm64"
  36. def execute(argv, *, log=None, **kwargs):
  37. if log:
  38. with Path(log).open("w") as output:
  39. return subprocess.run(argv, stdout=output, stderr=subprocess.STDOUT,
  40. check=True, **kwargs)
  41. return subprocess.run(argv, check=True, capture_output=True, text=True, **kwargs)
  42. def general_base_image():
  43. """Resolve the measured arm64 base independently of mutable Docker tags."""
  44. result = subprocess.run(["docker", "image", "inspect", GENERAL_IMAGE_ID], capture_output=True, text=True)
  45. if result.returncode:
  46. execute(["docker", "pull", "--platform", GENERAL_PLATFORM, GENERAL_IMAGE_DIGEST])
  47. result = execute(["docker", "image", "inspect", GENERAL_IMAGE_ID])
  48. image = json.loads(result.stdout)[0]
  49. if image["Id"] != GENERAL_IMAGE_ID or image["Architecture"] != "arm64":
  50. raise RuntimeError("General WorkBuddy runtime base does not match pinned arm64 image")
  51. return GENERAL_IMAGE_ID
  52. def fetch(url: str, destination: Path) -> bytes:
  53. if not destination.exists():
  54. destination.parent.mkdir(parents=True, exist_ok=True)
  55. temporary = destination.with_suffix(destination.suffix + ".part")
  56. with urllib.request.urlopen(url, timeout=90) as source, temporary.open("wb") as target:
  57. shutil.copyfileobj(source, target)
  58. temporary.replace(destination)
  59. return destination.read_bytes()
  60. def unpack(data: bytes, destination: Path, *, strip_package=False):
  61. destination.mkdir(parents=True, exist_ok=True)
  62. with tarfile.open(fileobj=io.BytesIO(data)) as archive:
  63. for member in archive.getmembers():
  64. if strip_package:
  65. member.name = member.name.partition("/")[2]
  66. if member.name:
  67. archive.extract(member, destination, filter="data")
  68. def npm(cache: Path, package: str, version: str, destination: Path):
  69. metadata = json.loads(fetch(
  70. f"https://registry.npmjs.org/{package}/{version}",
  71. cache / f"{destination.name}-{version}-metadata.json"))
  72. data = fetch(metadata["dist"]["tarball"], cache / f"{destination.name}-{version}.tgz")
  73. integrity = metadata["dist"]["integrity"]
  74. algorithm, expected = integrity.split("-", 1)
  75. actual = base64.b64encode(hashlib.new(algorithm, data).digest()).decode()
  76. if actual != expected:
  77. raise RuntimeError(f"Integrity mismatch for {package}@{version}")
  78. if not destination.exists():
  79. unpack(data, destination, strip_package=True)
  80. def prepare(cache: Path):
  81. cache.mkdir(parents=True, exist_ok=True)
  82. upstream = cache / "workbuddy-bench"
  83. if not upstream.exists():
  84. execute(["git", "clone", UPSTREAM, str(upstream)])
  85. revision = execute(["git", "-C", str(upstream), "rev-parse", "HEAD"]).stdout.strip()
  86. if revision != REVISION:
  87. raise RuntimeError(f"Expected upstream {REVISION}; found {revision}. Use a separate cache.")
  88. dataset_url = "https://huggingface.co/datasets/tencent/workbuddy-bench/resolve/main"
  89. data = fetch(f"{dataset_url}/{DATASET}.tar.gz", cache / "workbuddy-code.tar.gz")
  90. if hashlib.sha256(data).hexdigest() != DATA_SHA256:
  91. raise RuntimeError("Official Code dataset SHA256 mismatch")
  92. if not (upstream / "datasets" / DATASET).exists():
  93. unpack(data, upstream / "datasets")
  94. runtime = cache / "workbuddy-runtime"
  95. runtime.mkdir(exist_ok=True)
  96. npm(cache, "@openai/codex", f"{CODEX_VERSION}-linux-arm64", runtime / "codex")
  97. npm(cache, "oh-my-agent", OMA_VERSION, runtime / "oma")
  98. npm(cache, "jsonc-parser", "3.3.1", runtime / "oma/node_modules/jsonc-parser")
  99. if not (runtime / "node/bin/node").exists():
  100. filename = f"node-v{NODE_VERSION}-linux-arm64.tar.xz"
  101. origin = f"https://nodejs.org/dist/v{NODE_VERSION}"
  102. data = fetch(f"{origin}/{filename}", cache / filename)
  103. checksums = fetch(f"{origin}/SHASUMS256.txt", cache / "node-checksums.txt").decode()
  104. expected = next(line.split()[0] for line in checksums.splitlines() if line.endswith(filename))
  105. if hashlib.sha256(data).hexdigest() != expected:
  106. raise RuntimeError("Node SHA256 mismatch")
  107. unpack(data, runtime / "node", strip_package=True)
  108. if not (runtime / "cue").exists():
  109. data = fetch(f"https://github.com/cue-lang/cue/releases/download/v{CUE_VERSION}/"
  110. f"cue_v{CUE_VERSION}_linux_arm64.tar.gz", cache / "workbuddy-cue-linux-arm64.tar.gz")
  111. with tarfile.open(fileobj=io.BytesIO(data)) as archive:
  112. archive.extract("cue", runtime, filter="data")
  113. if not (runtime / "bun").exists():
  114. data = fetch(f"https://github.com/oven-sh/bun/releases/download/bun-v{BUN_VERSION}/"
  115. "bun-linux-aarch64.zip", cache / "workbuddy-bun-linux-arm64.zip")
  116. with zipfile.ZipFile(io.BytesIO(data)) as archive:
  117. (runtime / "bun").write_bytes(archive.read("bun-linux-aarch64/bun"))
  118. (runtime / "bun").chmod(0o755)
  119. venv = cache / "workbuddy-venv"
  120. if not (venv / "bin/python").exists():
  121. execute(["uv", "venv", str(venv), "--python", "3.12"])
  122. execute(["uv", "pip", "install", "--python", str(venv / "bin/python"),
  123. "--only-binary", ":all:", "harbor==0.18.0", "PyYAML"],
  124. log=cache / "workbuddy-host-dependencies.log")
  125. return upstream, runtime, venv
  126. def docker(container, *args, **kwargs):
  127. return execute(["docker", "exec", "-w", "/workspace", container, *args], **kwargs)
  128. def copy_into(container, source, target):
  129. execute(["docker", "cp", str(source), f"{container}:{target}"])
  130. def verifier_script(upstream: Path, venv: Path, output: Path):
  131. program = '''from pathlib import Path
  132. from workbuddy_bench.judge.registry import load_verifier_contract, load_verifier_registry, RegistryBuildContext
  133. from workbuddy_bench.judge.core import EvaluationContext
  134. import sys
  135. task = Path(sys.argv[1])
  136. contract = load_verifier_contract(task)
  137. registry = load_verifier_registry(RegistryBuildContext(contract=contract))
  138. context = EvaluationContext(dataset_id=contract.dataset_id, task_id=task.name, workspace='/workspace', tests_dir='/tests', verifier_dir='/logs/verifier')
  139. plan = registry.plan_builder(context)
  140. assert len(plan.judges) == 1 and plan.judges[0].type == 'rule_script'
  141. print('#!/bin/bash\\n' + plan.judges[0].config['command'])
  142. '''
  143. env = dict(os.environ, PYTHONPATH=str(upstream / "src"))
  144. task = upstream / "datasets" / DATASET / "tasks" / TASK
  145. result = execute([str(venv / "bin/python"), "-c", program, str(task)], env=env)
  146. output.write_text(result.stdout)
  147. def collect(container: str, upstream: Path, venv: Path, output: Path, *, baseline=True):
  148. task = upstream / "datasets" / DATASET / "tasks" / TASK
  149. status = docker(container, "cat", "/logs/agent/exit_code").stdout.strip()
  150. if status != "0":
  151. raise RuntimeError(f"Agent failed with exit code {status}; inspect its private logs")
  152. docker(container, "bash", "-lc", "mkdir -p /scorer /logs/verifier /tests; "
  153. "if [ ! -f /logs/verifier/agent.patch ]; then "
  154. "git add -A; git diff --staged > /logs/verifier/agent.patch; git reset >/dev/null; fi")
  155. copy_into(container, str(task / "tests") + "/.", "/tests/")
  156. copy_into(container, task / "environment/scorer.py", "/scorer/scorer.py")
  157. verifier_script(upstream, venv, output / "native-verifier.sh")
  158. copy_into(container, output / "native-verifier.sh", "/tmp/verify.sh")
  159. docker(container, "bash", "-lc", "PYTHONPATH=/workspace bash /tmp/verify.sh",
  160. log=output / "native-verifier.log")
  161. execute(["docker", "cp", f"{container}:/logs", str(output)])
  162. if baseline:
  163. # Only restore the production file; keep the same injected official tests.
  164. docker(container, "bash", "-lc", "rm -rf /logs/baseline-verifier; "
  165. "cp asgiref/local.py /tmp/agent-local.py; "
  166. "git show HEAD:asgiref/local.py > asgiref/local.py; "
  167. "mv /logs/verifier /logs/agent-verifier; mkdir /logs/verifier; "
  168. ": > /logs/verifier/agent.patch; "
  169. "PYTHONPATH=/workspace bash /tmp/verify.sh > /logs/verifier/native-verifier.log 2>&1; "
  170. "result=$?; mv /logs/verifier /logs/baseline-verifier; "
  171. "mv /logs/agent-verifier /logs/verifier; cp /tmp/agent-local.py asgiref/local.py; exit $result")
  172. execute(["docker", "cp", f"{container}:/logs/baseline-verifier", str(output)])
  173. metadata = docker(container, "python", "-c", '''from pathlib import Path
  174. import hashlib, importlib.metadata, json, platform
  175. result = {}
  176. for path in Path('/root/.codex/sessions').rglob('*.jsonl'):
  177. for line in path.read_text().splitlines():
  178. event = json.loads(line)
  179. if event.get('type') == 'turn_context':
  180. result['model'] = event['payload'].get('model')
  181. break
  182. result['skills'] = json.loads(Path('/workspace/.agents/skills/_version.json').read_text())
  183. result['python'] = platform.python_version()
  184. result['packages'] = {name: importlib.metadata.version(name) for name in ['pytest', 'pytest-asyncio', 'serena-agent']}
  185. result['oma_debug_skill_sha256'] = hashlib.sha256(Path('/workspace/.agents/skills/oma-debug/SKILL.md').read_bytes()).hexdigest()
  186. result['agents_md_sha256'] = hashlib.sha256(Path('/workspace/AGENTS.md').read_bytes()).hexdigest()
  187. print(json.dumps(result))
  188. ''').stdout
  189. score = json.loads((output / "logs/verifier/score.json").read_text())
  190. trajectory = [json.loads(line) for line in (output / "logs/agent/trajectory.jsonl").read_text().splitlines()]
  191. usage = next((event.get("usage") for event in reversed(trajectory) if event.get("type") == "turn.completed"), None)
  192. source_files = [task / "instruction.md", task / "environment/Dockerfile",
  193. task / "environment/scorer.py", task / "tests/verifier.toml",
  194. task / "tests/gold.patch",
  195. upstream / "datasets" / DATASET / "shared/verifier/rule.py",
  196. upstream / "datasets" / DATASET / "shared/verifier/plugin.py"]
  197. source_files += sorted((task / "tests/injected").rglob("*.py"))
  198. source_hashes = {str(path.relative_to(upstream)): hashlib.sha256(path.read_bytes()).hexdigest()
  199. for path in source_files if path.is_file()}
  200. with tarfile.open(upstream.parent / "workbuddy-code.tar.gz") as archive:
  201. for path, expected in source_hashes.items():
  202. original = archive.extractfile(path.removeprefix("datasets/"))
  203. if original is None or hashlib.sha256(original.read()).hexdigest() != expected:
  204. raise RuntimeError(f"Verifier source differs from verified dataset: {path}")
  205. base_image = json.loads(execute(["docker", "image", "inspect", GENERAL_IMAGE_ID]).stdout)[0]
  206. receipt = dict(status="completed", mode="oma-config-and-skills-smoke", task=TASK,
  207. upstream_revision=REVISION, dataset_sha256=DATA_SHA256,
  208. oma_cli_version=OMA_VERSION, codex_version=CODEX_VERSION,
  209. hooks_verified=False, agent_exit_code=0, score=score, usage=usage,
  210. runtime=json.loads(metadata), canonical_harbor_run=False,
  211. base_image_id=base_image["Id"], source_hashes=source_hashes,
  212. verifier_sources_match_verified_archive=True,
  213. environment_overrides=["arm64 prebuilt base; runtime package installation",
  214. "Node 24.11.1 arm64 instead of 20.18.1 x64",
  215. "PYTHONPATH source imports instead of editable package build",
  216. "Codex ChatGPT OAuth instead of an API-proxy harness image"],
  217. verifier_sha256=hashlib.sha256((output / "native-verifier.sh").read_bytes()).hexdigest())
  218. if baseline:
  219. receipt["original_source_score"] = json.loads((output / "baseline-verifier/score.json").read_text())
  220. (output / "receipt.json").write_text(json.dumps(receipt, indent=2) + "\n")
  221. print(json.dumps(receipt, indent=2))
  222. def main():
  223. parser = argparse.ArgumentParser(description=__doc__)
  224. parser.add_argument("mode", choices=["prepare", "run", "collect"], nargs="?", default="run")
  225. parser.add_argument("--cache", type=Path, default=Path.home() / ".cache/oma-benchmarks")
  226. parser.add_argument("--output", type=Path)
  227. parser.add_argument("--container", help="Existing completed smoke container for collect")
  228. parser.add_argument("--timeout-sec", type=int, default=900)
  229. parser.add_argument("--keep-container", action="store_true")
  230. args = parser.parse_args()
  231. if platform.machine() not in ("arm64", "aarch64"):
  232. parser.error("This pinned smoke runner currently supports arm64 Docker hosts")
  233. upstream, runtime, venv = prepare(args.cache.resolve())
  234. if args.mode == "prepare":
  235. print(f"Prepared {TASK}; no agent was run")
  236. return
  237. output = (args.output or args.cache / "workbuddy-results" / time.strftime("%Y%m%d-%H%M%S")).resolve()
  238. output.mkdir(parents=True, exist_ok=True)
  239. if args.mode == "collect":
  240. if not args.container:
  241. parser.error("collect requires --container")
  242. collect(args.container, upstream, venv, output)
  243. return
  244. auth = Path(os.environ.get("CODEX_HOME", str(Path.home() / ".codex"))) / "auth.json"
  245. if not auth.is_file():
  246. parser.error("Codex auth.json not found; sign in with codex login first")
  247. container = f"oma-workbuddy-{os.getpid()}-{int(time.time())}"
  248. execute(["docker", "run", "-d", "--platform", GENERAL_PLATFORM, "--name", container, "--cpus", "2", "--memory", "4g",
  249. "--mount", f"type=bind,source={runtime},target=/opt/prebuilt,readonly",
  250. "--mount", f"type=bind,source={auth},target=/root/.codex/auth.json,readonly",
  251. "-e", "CODEX_HOME=/root/.codex", "-e", "CI=true", "-e", "OMA_SKIP_VERSION_CHECK=1",
  252. general_base_image(), "sleep", "infinity"])
  253. try:
  254. task = upstream / "datasets" / DATASET / "tasks" / TASK
  255. copy_into(container, task / "environment/workspace.tar.gz", "/tmp/workspace.tar.gz")
  256. execute(["docker", "exec", container, "mkdir", "-p", "/workspace"])
  257. docker(container, "bash", "-lc", '''set -e
  258. mkdir -p /workspace /logs/agent /logs/verifier
  259. tar xzf /tmp/workspace.tar.gz -C /workspace
  260. rm /tmp/workspace.tar.gz
  261. export DEBIAN_FRONTEND=noninteractive
  262. apt-get update -qq
  263. apt-get install -y -qq --no-install-recommends git curl bash ca-certificates xz-utils procps
  264. python -m pip install --only-binary=:all: pytest pytest-asyncio 'serena-agent==1.0.0'
  265. ln -s /opt/prebuilt/node/bin/node /usr/local/bin/node
  266. ln -s /opt/prebuilt/bun /usr/local/bin/bun
  267. ln -s /opt/prebuilt/cue /usr/local/bin/cue
  268. ln -s /opt/prebuilt/codex/vendor/aarch64-unknown-linux-musl/bin/codex /usr/local/bin/codex
  269. ln -s /opt/prebuilt/codex/vendor/aarch64-unknown-linux-musl/codex-path/rg /usr/local/bin/rg
  270. ln -s /opt/prebuilt/oma/bin/cli.js /usr/local/bin/oma
  271. cd /workspace
  272. git config --global user.email dev@project
  273. git config --global user.name Developer
  274. git init -q -b main
  275. git add -A
  276. git commit -q -m 'initial setup'
  277. codex login status
  278. oma -y install --semantic-memory none
  279. oma link codex
  280. printf '[projects."/workspace"]\ntrust_level = "trusted"\n' > /root/.codex/config.toml
  281. git add -A
  282. git commit -q -m 'Install OMA harness overlay'
  283. ''', log=output / "setup.log")
  284. prompt = ("Use the installed OMA debug skill for this task and follow the workspace instructions. "
  285. "Do not start a workflow. Never build, compile, bundle or package software. "
  286. "The Python source is importable from /workspace; pytest and pytest-asyncio are already installed. "
  287. "Implement and test the requested fix.\n\n" + (task / "instruction.md").read_text())
  288. (output / "prompt.txt").write_text(prompt)
  289. copy_into(container, output / "prompt.txt", "/tmp/prompt.txt")
  290. docker(container, "bash", "-lc", '''codex exec --dangerously-bypass-approvals-and-sandbox --json --output-last-message /logs/agent/final.md - < /tmp/prompt.txt > /logs/agent/trajectory.jsonl 2> /logs/agent/stderr.log
  291. result=$?
  292. printf '%s\n' "$result" > /logs/agent/exit_code
  293. exit "$result"
  294. ''', timeout=args.timeout_sec, log=output / "agent-launch.log")
  295. collect(container, upstream, venv, output)
  296. finally:
  297. if args.keep_container:
  298. print(f"Container retained: {container}")
  299. else:
  300. execute(["docker", "rm", "-f", container])
  301. if __name__ == "__main__":
  302. main()
benchmarks/external/workbuddy/suite.py · Bundled script · 731 lines · 1 finding marked
  1. #!/usr/bin/env python3
  2. """Sequential, resumable traversal of all 260 official WorkBuddy tasks."""
  3. from __future__ import annotations
  4. import argparse
  5. import collections
  6. import fcntl
  7. import hashlib
  8. import json
  9. import math
  10. import os
  11. from pathlib import Path
  12. import re
  13. import shlex
  14. import shutil
  15. import subprocess
  16. import sys
  17. import time
  18. import tomllib
  19. import traceback
  20. import zipfile
  21. import run as smoke
  22. SUBSETS = {"code": 80, "web": 70, "office": 50, "sec": 60}
  23. CHECKSUMS = {"code": smoke.DATA_SHA256,
  24. "web": "bf4e0258501a7007d547eeb5f3e9cfb7dd648feaaa4f3dbc23dbae87758e0b66",
  25. "office": "89d4480b483ca2e0555ba96cc7d73c16c78802b17cd822044d5f7f27ab0558bf",
  26. "sec": "f615f55b2ce68294eca6bef658d3a135978ca1e7f00b8e712292a2738f79c3f3"}
  27. def frozen_oma_seed(cache):
  28. seed = cache / "workbuddy-security-oma-seed"
  29. version = json.loads((seed / "skills/_version.json").read_text())["version"]
  30. if version != "20.10.3":
  31. raise RuntimeError(f"Expected frozen OMA 20.10.3 snapshot; found {version}")
  32. return seed
  33. def version_scaffold(task, cache):
  34. """Use published wheel metadata where the official editable install generates it."""
  35. text = (Path(task["path"]) / "environment/Dockerfile").read_text()
  36. if "SETUPTOOLS_SCM_PRETEND_VERSION_FOR_PYTEST=8.4.2" not in text:
  37. return None
  38. metadata = json.loads(smoke.fetch("https://pypi.org/pypi/pytest/8.4.2/json",
  39. cache / "workbuddy-pytest-8.4.2-metadata.json"))
  40. wheel = next(item for item in metadata["urls"] if item["filename"] == "pytest-8.4.2-py3-none-any.whl")
  41. archive = cache / wheel["filename"]
  42. data = smoke.fetch(wheel["url"], archive)
  43. if hashlib.sha256(data).hexdigest() != wheel["digests"]["sha256"]:
  44. raise RuntimeError("Published pytest wheel checksum mismatch")
  45. target = cache / "workbuddy-pytest-8.4.2-version.py"
  46. with zipfile.ZipFile(archive) as contents:
  47. target.write_bytes(contents.read("_pytest/_version.py"))
  48. return {"source": "official-pytest-8.4.2-wheel", "path": str(target),
  49. "destination": "src/_pytest/_version.py", "sha256": hashlib.sha256(target.read_bytes()).hexdigest()}
  50. def write_json(path, value):
  51. path.parent.mkdir(parents=True, exist_ok=True)
  52. temporary = path.with_suffix(".tmp")
  53. temporary.write_text(json.dumps(value, indent=2, ensure_ascii=False) + "\n")
  54. temporary.replace(path)
  55. def prepare(args):
  56. upstream, runtime, venv = smoke.prepare(args.cache)
  57. for subset, count in SUBSETS.items():
  58. dataset = f"wb-bench-{subset}-v1.0"
  59. archive = args.cache / f"workbuddy-{subset}.tar.gz"
  60. data = smoke.fetch(f"https://huggingface.co/datasets/tencent/workbuddy-bench/resolve/main/{dataset}.tar.gz", archive)
  61. if hashlib.sha256(data).hexdigest() != CHECKSUMS[subset]:
  62. raise RuntimeError(f"Dataset checksum mismatch: {subset}")
  63. if not (upstream / "datasets" / dataset).exists():
  64. smoke.unpack(data, upstream / "datasets")
  65. actual = list((upstream / "datasets" / dataset / "tasks").glob("*/task.toml"))
  66. if len(actual) != count:
  67. raise RuntimeError(f"Expected {count} {subset} tasks, found {len(actual)}")
  68. return upstream, runtime, venv
  69. def inventory(upstream):
  70. tasks = []
  71. for subset in SUBSETS:
  72. root = upstream / "datasets" / f"wb-bench-{subset}-v1.0" / "tasks"
  73. for directory in sorted(root.iterdir()):
  74. config = tomllib.loads((directory / "task.toml").read_text())
  75. dockerfile = (directory / "environment/Dockerfile").read_text()
  76. tasks.append(dict(id=f"{subset}/{directory.name}", subset=subset,
  77. task=directory.name, path=str(directory),
  78. status="pending", score=None,
  79. dockerfile_sha256=hashlib.sha256(dockerfile.encode()).hexdigest(),
  80. workspace_archive=(directory / "environment/workspace.tar.gz").is_file(),
  81. needs_task_image=(subset == "sec"),
  82. timeout_sec=config.get("agent", {}).get("timeout_sec", 1800),
  83. metadata=config.get("metadata", {})))
  84. return tasks
  85. def d_exec(container, command, *, log=None, timeout=None, check=True, cwd="/workspace"):
  86. argv = ["docker", "exec", "-w", cwd, container, "bash", "-lc", command]
  87. if log:
  88. with Path(log).open("w") as output:
  89. return subprocess.run(argv, stdout=output, stderr=subprocess.STDOUT,
  90. timeout=timeout, check=check)
  91. return subprocess.run(argv, capture_output=True, text=True, timeout=timeout, check=check)
  92. DEPENDENCIES = r'''from pathlib import Path
  93. import ast, configparser, json, re, shlex, tomllib
  94. from packaging.requirements import Requirement, InvalidRequirement
  95. requirements = []
  96. setup_metadata = []
  97. runtime_compatibility = []
  98. local_extras = set()
  99. local_no_deps = False
  100. local_paths={str(Path('.').resolve()),str(Path('src').resolve())}
  101. local_versions={}
  102. lockfile=Path('poetry.lock')
  103. lock_versions={item['name']:item['version'] for item in tomllib.loads(lockfile.read_text()).get('package',[])} if lockfile.is_file() else {}
  104. def remember_local_extras(token):
  105. match=re.fullmatch(r'(?:\.|\./[^\[]*|/workspace[^\[]*)\[([^\]]+)\]',token)
  106. if match: local_extras.update(x.strip() for x in match.group(1).split(','))
  107. def remember_local_project(token):
  108. value=token.split('[',1)[0]
  109. if value.startswith(('./','/workspace')) or value=='.':
  110. path=Path(value).resolve()
  111. if path.is_dir(): local_paths.add(str(path))
  112. dockerfile = Path('/tmp/task-Dockerfile').read_text().replace('\\\n', ' ')
  113. for line in dockerfile.splitlines():
  114. if not line.lstrip().startswith('RUN '): continue
  115. try:
  116. command=line.lstrip()[4:]
  117. outer=shlex.split(command)
  118. if len(outer)==3 and outer[0] in ['bash','sh'] and outer[1] in ['-c','-lc']: command=outer[2]
  119. lexer=shlex.shlex(command,posix=True,punctuation_chars=';&|')
  120. lexer.whitespace_split=True;lexer.commenters=''
  121. tokens=list(lexer)
  122. except ValueError: continue
  123. clauses=[[]]
  124. for token in tokens:
  125. if token in ['&&','||',';','|']: clauses.append([])
  126. else: clauses[-1].append(token)
  127. for clause in clauses:
  128. args=None
  129. for i,token in enumerate(clause):
  130. if token in ['pip','pip3'] and clause[i+1:i+2]==['install']: args=clause[i+2:];break
  131. if token.startswith('python') and clause[i+1:i+4]==['-m','pip','install']: args=clause[i+4:];break
  132. if args is None: continue
  133. if '--no-deps' in args and any(token in ['-e','--editable'] and i+1<len(args) and args[i+1].startswith(('.','/workspace')) for i,token in enumerate(args)):
  134. local_no_deps=True
  135. i=0
  136. while i<len(args):
  137. token=args[i];i+=1
  138. if token in ['-r','--requirement','-c','--constraint']:
  139. if i<len(args):
  140. path=Path(args[i]).resolve();i+=1
  141. if not path.is_file(): raise RuntimeError(f'Missing official dependency file: {path}')
  142. requirements.append(('-c ' if token in ['-c','--constraint'] else '-r ')+str(path))
  143. continue
  144. if token in ['-e','--editable']:
  145. if i<len(args): remember_local_extras(args[i]);remember_local_project(args[i])
  146. i+=1;continue
  147. remember_local_extras(token)
  148. remember_local_project(token)
  149. if token in ['--index-url','--extra-index-url','--find-links','--trusted-host']:
  150. i+=1;continue
  151. if token.startswith('-') or token.startswith(('./','/','git+','http:','https:','.[')) or token=='.': continue
  152. locked=re.fullmatch(r'([A-Za-z0-9_.-]+)==\$(?:\{[A-Za-z_][A-Za-z0-9_]*\}|[A-Za-z_][A-Za-z0-9_]*)',token)
  153. if locked:
  154. name=locked.group(1)
  155. if name not in lock_versions: raise RuntimeError(f'Cannot resolve official locked dependency: {token}')
  156. token=name+'=='+lock_versions[name]
  157. runtime_compatibility.append({'source':'official Dockerfile and poetry.lock','requirement':token})
  158. try: Requirement(token)
  159. except InvalidRequirement: continue
  160. requirements.append(token)
  161. for base in map(Path,sorted(local_paths)):
  162. p=base/'pyproject.toml'
  163. if p.exists():
  164. try:
  165. data=tomllib.loads(p.read_text());project=data.get('project',{})
  166. if project.get('name') and project.get('version'): local_versions[re.sub(r'[-_.]+','-',project['name']).lower()]=project['version']
  167. requirements.extend(data.get('build-system',{}).get('requires',[]))
  168. if not local_no_deps: requirements.extend(project.get('dependencies',[]))
  169. for name, deps in project.get('optional-dependencies',{}).items():
  170. if name in local_extras and not local_no_deps: requirements.extend(deps)
  171. if project.get('name')=='fastapi':
  172. for dependency in project.get('optional-dependencies',{}).get('test',[]):
  173. if Requirement(dependency).name=='anyio':
  174. requirements.append(dependency)
  175. runtime_compatibility.append({'source':'project.optional-dependencies.test','requirement':dependency,'reason':'Use the declared test backend with the legacy Starlette API'})
  176. except Exception: pass
  177. p=base/'setup.cfg'
  178. if p.exists():
  179. try:
  180. cfg=configparser.ConfigParser(interpolation=None);cfg.read(p)
  181. requirements.extend(cfg.get('options','install_requires',fallback='').splitlines())
  182. for key in local_extras:
  183. requirements.extend(cfg.get('options.extras_require',key,fallback='').splitlines())
  184. except Exception: pass
  185. p=base/'setup.py'
  186. if p.is_file():
  187. for node in ast.walk(ast.parse(p.read_text())):
  188. if not isinstance(node,ast.Call) or not (isinstance(node.func,ast.Name) and node.func.id=='setup' or isinstance(node.func,ast.Attribute) and node.func.attr=='setup'): continue
  189. metadata={}
  190. for keyword in node.keywords:
  191. if keyword.arg not in ['install_requires','extras_require','entry_points']: continue
  192. try: metadata[keyword.arg]=ast.literal_eval(keyword.value)
  193. except (ValueError,TypeError): raise RuntimeError(f'Unsupported nonliteral setup.py metadata: {p}:{keyword.arg}')
  194. requirements.extend(metadata.get('install_requires',[]))
  195. for name,deps in metadata.get('extras_require',{}).items():
  196. if name in local_extras: requirements.extend(deps)
  197. setup_metadata.append(metadata)
  198. requirements = sorted(set(x.strip() for x in requirements if x.strip() and not x.lstrip().startswith(('#','.','git+','http:','https:'))))
  199. debian_packages=[]
  200. for dependency in list(requirements):
  201. if dependency.startswith(('-r ','-c ')): continue
  202. parsed=Requirement(dependency)
  203. name=re.sub(r'[-_.]+','-',parsed.name).lower()
  204. if name in local_versions:
  205. if not parsed.specifier.contains(local_versions[name],prereleases=True): raise RuntimeError(f'Incompatible declared local project: {dependency}')
  206. requirements.remove(dependency);continue
  207. if parsed.name=='httpretty':
  208. requirements.remove(dependency);debian_packages.append('python3-httpretty')
  209. runtime_compatibility.append({'source':'official Dockerfile dependency via prebuilt Debian package','requirement':dependency,'package':'python3-httpretty'})
  210. Path('/tmp/task-requirements.txt').write_text('\n'.join(requirements)+'\n')
  211. Path('/tmp/task-debian-python.txt').write_text('\n'.join(debian_packages)+ ('\n' if debian_packages else ''))
  212. Path('/tmp/task-setup-metadata.json').write_text(json.dumps(setup_metadata))
  213. Path('/tmp/task-runtime-compatibility.json').write_text(json.dumps(runtime_compatibility))
  214. Path('/tmp/task-local-projects.json').write_text(json.dumps(sorted(local_paths)))
  215. print('Official editable extras:',sorted(local_extras))
  216. '''
  217. PROJECT_ENTRYPOINTS = r'''from pathlib import Path
  218. import configparser,json,re,sysconfig,tomllib
  219. scripts={}
  220. project_metadata=[]
  221. source_paths=[]
  222. roots=Path('/tmp/task-local-projects.json')
  223. for base in map(Path,json.loads(roots.read_text()) if roots.is_file() else ['.','src']):
  224. config=base/'pyproject.toml'
  225. if not config.is_file(): continue
  226. project=tomllib.loads(config.read_text()).get('project',{})
  227. scripts.update(project.get('scripts',{}))
  228. name,version=project.get('name'),project.get('version')
  229. if name and version:
  230. distribution=Path(sysconfig.get_paths()['purelib'])/(re.sub(r'[-_.]+','_',name)+'-'+version+'.dist-info')
  231. distribution.mkdir(exist_ok=True)
  232. lines=['Metadata-Version: 2.4','Name: '+name,'Version: '+version]
  233. if project.get('requires-python'): lines.append('Requires-Python: '+project['requires-python'])
  234. lines.extend('Requires-Dist: '+value for value in project.get('dependencies',[]))
  235. (distribution/'METADATA').write_text('\n'.join(lines)+'\n\n')
  236. groups=dict(project.get('entry-points',{}))
  237. if project.get('scripts'): groups['console_scripts']=project['scripts']
  238. entry_lines=[]
  239. for group,entries in groups.items():
  240. entry_lines.append('['+group+']')
  241. entry_lines.extend(key+' = '+value for key,value in entries.items())
  242. if entry_lines:(distribution/'entry_points.txt').write_text('\n'.join(entry_lines)+'\n')
  243. for path in [base,base/'src']:
  244. if path.is_dir(): source_paths.append(str(path.resolve()))
  245. project_metadata.append({'source':'static project metadata in pyproject.toml','name':name,'version':version,'path':str(distribution),'entry_point_groups':list(groups)})
  246. metadata=Path('/tmp/task-setup-metadata.json')
  247. for setup in json.loads(metadata.read_text()) if metadata.is_file() else []:
  248. points=setup.get('entry_points',{})
  249. if isinstance(points,str):
  250. parser=configparser.ConfigParser(interpolation=None);parser.optionxform=str;parser.read_string(points)
  251. scripts.update(dict(parser.items('console_scripts')) if parser.has_section('console_scripts') else {})
  252. else:
  253. for line in points.get('console_scripts',[]):
  254. name,separator,target=line.partition('=')
  255. if separator:scripts[name.strip()]=target.strip()
  256. for name,target in scripts.items():
  257. if not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9._-]*',name): raise ValueError(name)
  258. module,separator,attribute=target.partition(':')
  259. if not separator: raise ValueError(target)
  260. launcher=Path('/usr/local/bin')/name
  261. launcher.write_text('#!/usr/local/bin/python\nimport importlib,sys\n'
  262. +f'entry=importlib.import_module({module!r})\n'
  263. +f'for part in {attribute!r}.split("."): entry=getattr(entry,part)\n'
  264. +'sys.exit(entry())\n')
  265. launcher.chmod(0o755)
  266. print('Installed declared source console entry point:',name,target)
  267. Path('/tmp/task-project-metadata.json').write_text(json.dumps(project_metadata))
  268. if source_paths:(Path(sysconfig.get_paths()['purelib'])/'oma-task-source-projects.pth').write_text('\n'.join(sorted(set(source_paths)))+'\n')
  269. '''
  270. def compatibility_setup(dockerfile: str) -> str:
  271. """Keep official Python compatibility/file scaffolds without package builds."""
  272. commands = []
  273. for line in dockerfile.replace("\\\n", " ").splitlines():
  274. line = line.lstrip()
  275. if line.startswith("RUN mkdir -p /opt/bench-agent-pack"):
  276. commands.append(line[4:])
  277. continue
  278. if not line.startswith("RUN "):
  279. continue
  280. lexer = shlex.shlex(line[4:], posix=True, punctuation_chars=";&|")
  281. lexer.whitespace_split = True
  282. tokens = list(lexer)
  283. clauses = [[]]
  284. for token in tokens:
  285. if token in {"&&", ";", "||", "|"}:
  286. clauses.append([])
  287. else:
  288. clauses[-1].append(token)
  289. for clause in clauses:
  290. if clause[:2] in [["python", "-c"], ["python3", "-c"]]:
  291. commands.append(shlex.join(clause))
  292. return "#!/bin/bash\nset -euo pipefail\n" + "\n".join(commands) + "\n"
  293. def copy_public_environment(container, directory):
  294. """Honor public Docker COPY inputs; task tests remain unavailable to the solver."""
  295. text = (directory / "Dockerfile").read_text().replace("\\\n", " ")
  296. for line in text.splitlines():
  297. line = line.strip()
  298. if not line.startswith("COPY ") or "--from=" in line:
  299. continue
  300. arguments = shlex.split(line[5:])
  301. if len(arguments) != 2:
  302. raise RuntimeError(f"Unsupported public task COPY: {line}")
  303. source, target = arguments
  304. if source == "workspace.tar.gz":
  305. continue
  306. public = directory / source
  307. if public.is_dir():
  308. d_exec(container, f"mkdir -p {shlex.quote(target)}")
  309. smoke.copy_into(container, str(public) + "/.", target.rstrip("/") + "/")
  310. elif public.is_file():
  311. d_exec(container, f"mkdir -p {shlex.quote(str(Path(target).parent))}")
  312. smoke.copy_into(container, public, target)
  313. else:
  314. raise RuntimeError(f"Public task environment input missing: {public}")
  315. MCP_FIXTURE_SETUP = r'''from pathlib import Path
  316. import json
  317. source=Path('/root/.claude.json')
  318. if source.exists():
  319. config=json.loads(source.read_text()); servers=config.get('mcpServers',{})
  320. if servers:
  321. with Path('/workspace/.codex/config.toml').open('a') as out:
  322. for name,server in servers.items():
  323. if not server.get('command'): continue
  324. out.write('\n[mcp_servers.'+json.dumps(name)+']\n')
  325. out.write('command = '+json.dumps(server['command'])+'\n')
  326. out.write('args = '+json.dumps(server.get('args',[]))+'\n')
  327. if server.get('env'):
  328. out.write('[mcp_servers.'+json.dumps(name)+'.env]\n')
  329. for key,value in server['env'].items():out.write(json.dumps(key)+' = '+json.dumps(str(value))+'\n')
  330. skill=Path('/opt/bench-agent-pack/agent_pack/skills/recruiting_search')
  331. target=Path('/workspace/.codex/skills/recruiting_search')
  332. if skill.is_dir():
  333. target.parent.mkdir(parents=True,exist_ok=True)
  334. if not target.exists() and not target.is_symlink():target.symlink_to(skill)
  335. '''
  336. BOOTSTRAP = r'''set -euo pipefail
  337. export DEBIAN_FRONTEND=noninteractive
  338. mkdir -p /workspace /logs/agent /logs/verifier /root/.codex
  339. apt-get update -qq
  340. apt-get install -y -qq --no-install-recommends git curl bash ca-certificates xz-utils procps
  341. ln -sf /opt/prebuilt/node/bin/node /usr/local/bin/node
  342. ln -sf /opt/prebuilt/node/bin/npm /usr/local/bin/npm
  343. ln -sf /opt/prebuilt/node/bin/npx /usr/local/bin/npx
  344. ln -sf /opt/prebuilt/bun /usr/local/bin/bun
  345. ln -sf /opt/prebuilt/cue /usr/local/bin/cue
  346. ln -sf /opt/prebuilt/codex/vendor/aarch64-unknown-linux-musl/bin/codex /usr/local/bin/codex
  347. ln -sf /opt/prebuilt/codex/vendor/aarch64-unknown-linux-musl/codex-path/rg /usr/local/bin/rg
  348. ln -sf /opt/prebuilt/oma/bin/cli.js /usr/local/bin/oma
  349. python -m venv /opt/oma-tooling
  350. /opt/oma-tooling/bin/python -m pip install --only-binary=:all: 'serena-agent==1.0.0'
  351. printf '#!/bin/bash\nexec env -u PYTHONPATH /opt/oma-tooling/bin/serena "$@"\n' > /usr/local/bin/serena
  352. chmod +x /usr/local/bin/serena
  353. python -m pip install --only-binary=:all: pytest
  354. cd /workspace
  355. if [ -f /tmp/workspace.tar.gz ]; then tar xzf /tmp/workspace.tar.gz -C /workspace; rm /tmp/workspace.tar.gz; fi
  356. bash /tmp/task-compat-setup.sh
  357. if [ -f /tmp/task-pytest-version.py ]; then cp /tmp/task-pytest-version.py src/_pytest/_version.py; fi
  358. python /tmp/derive-dependencies.py
  359. if [ -s /tmp/task-debian-python.txt ]; then
  360. xargs -r apt-get install -y -qq --no-install-recommends < /tmp/task-debian-python.txt
  361. python - <<'PY'
  362. from pathlib import Path
  363. import json,subprocess,sysconfig
  364. destination=Path(sysconfig.get_paths()['purelib'])
  365. for source in Path('/usr/lib/python3/dist-packages').glob('httpretty*'):
  366. target=destination/source.name
  367. if not target.exists(): target.symlink_to(source)
  368. path=Path('/tmp/task-runtime-compatibility.json')
  369. conditions=json.loads(path.read_text())
  370. for condition in conditions:
  371. if condition.get('package')=='python3-httpretty':
  372. condition['installed_version']=subprocess.check_output(['dpkg-query','-W','-f','${Version}','python3-httpretty'],text=True).strip()
  373. path.write_text(json.dumps(conditions))
  374. PY
  375. fi
  376. if [ -s /tmp/task-requirements.txt ]; then python -m pip install --only-binary=:all: -r /tmp/task-requirements.txt; fi
  377. python /tmp/install-task-entrypoints.py
  378. bash /tmp/task-postinstall-fixtures.sh
  379. git config --global user.email dev@project
  380. git config --global user.name Developer
  381. git init -q -b main
  382. git add -A
  383. git -c core.hooksPath=/dev/null commit -q --allow-empty -m 'initial task workspace'
  384. oma link codex
  385. python /tmp/project-fixture-mcp.py
  386. printf '[projects."/workspace"]\ntrust_level = "trusted"\n' > /root/.codex/config.toml
  387. git add -A
  388. git -c core.hooksPath=/dev/null commit -q --allow-empty -m 'Install OMA harness overlay'
  389. codex login status
  390. '''
  391. def result_status(output, subset):
  392. path = output / "verifier/score.json"
  393. reward_path = output / "verifier/reward.json"
  394. score = json.loads(path.read_text()) if path.exists() else {}
  395. reward = json.loads(reward_path.read_text()) if reward_path.exists() else {}
  396. if not score and not reward and (output / "verifier/reward.txt").exists():
  397. try:
  398. reward = {"reward": float((output / "verifier/reward.txt").read_text().strip()),
  399. "source": "native-reward.txt"}
  400. except ValueError:
  401. pass
  402. if not score and not reward:
  403. return "verifier_error", None, {"error": "Official verifier produced no score/reward"}
  404. test_status = str(score.get("test_status", "")).lower()
  405. if test_status in {"build_error", "judge_error"}:
  406. return "environment_error" if test_status == "build_error" else "judge_error", None, score
  407. errors = score.get("diagnostics", {}).get("status_counts", {})
  408. if errors.get("build_error", 0) > 0:
  409. return "environment_error", None, score
  410. if errors.get("judge_error", 0) > 0:
  411. return "judge_error", None, score
  412. value = score.get("overall", score.get("reward", score.get("test_pass_rate")))
  413. if value is None:
  414. value = reward.get("overall", reward.get("reward", reward.get("test_pass_rate")))
  415. if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value) or not 0 <= value <= 1:
  416. return "verifier_error", None, score or reward
  417. return "scored", float(value), score or reward
  418. def agent_metadata(container, workspace="/workspace", codex_home="/root/.codex"):
  419. program = r'''from pathlib import Path
  420. import hashlib,json,sys
  421. workspace=Path(sys.argv[1]); codex_home=Path(sys.argv[2])
  422. result={}
  423. for line in Path('/logs/agent/trajectory.jsonl').read_text().splitlines():
  424. try: event=json.loads(line)
  425. except ValueError: continue
  426. if event.get('type')=='turn.completed': result['usage']=event.get('usage')
  427. if event.get('type')=='turn.failed': result['failure']=event.get('error')
  428. for path in (codex_home/'sessions').rglob('*.jsonl'):
  429. for line in path.read_text().splitlines():
  430. try: event=json.loads(line)
  431. except ValueError: continue
  432. if event.get('type')=='turn_context':
  433. result['actual_model']=event.get('payload',{}).get('model')
  434. result['actual_effort']=event.get('payload',{}).get('effort')
  435. break
  436. result['skills']=json.loads((workspace/'.agents/skills/_version.json').read_text())
  437. for name,path in [('agents_md_sha256',workspace/'AGENTS.md'),('debug_skill_sha256',workspace/'.agents/skills/oma-debug/SKILL.md')]:
  438. result[name]=hashlib.sha256(Path(path).read_bytes()).hexdigest()
  439. print(json.dumps(result))
  440. '''
  441. captured = subprocess.run(["docker", "exec", container, "python3", "-c", program, workspace, codex_home], capture_output=True, text=True)
  442. return json.loads(captured.stdout) if captured.returncode == 0 else {"metadata_error": captured.stderr[-400:]}
  443. def run_task(task, args, upstream, runtime, venv):
  444. path = Path(task["path"])
  445. output = args.output / task["subset"] / task["task"]
  446. if (output / "status.json").exists():
  447. previous = output.with_name(output.name + ".attempts") / str(time.time_ns())
  448. previous.parent.mkdir(parents=True, exist_ok=True)
  449. shutil.move(str(output), previous)
  450. output.mkdir(parents=True, exist_ok=True)
  451. stale = {"status", "score", "output", "reason", "container", "verifier", "agent_runtime", "started_at", "finished_at",
  452. "base_image_id", "platform", "judge_condition", "judge_model", "judge_effort", "regrade_of",
  453. "model_invoked", "environment_scaffolds", "saved_agent_patch_sha256", "tooling_python_isolated", "runtime_compatibility", "npm_global_prefix", "source_distribution_metadata", "dependency_download_timeout_sec"}
  454. result = {key: value for key, value in task.items() if key not in stale}
  455. result.update(status="preparing", score=None, output=str(output), started_at=time.time())
  456. result.update(requested_model="gpt-6.1-sol", requested_effort="high", oma_definitions_version="20.10.3")
  457. write_json(output / "status.json", result)
  458. if task["subset"] == "sec":
  459. return dict(result, status="waiting_provider_access", score=None,
  460. reason="Retained in inventory; execution requires appropriate provider access")
  461. if task["subset"] == "web" and not args.judge_config:
  462. # Keep in inventory, but don't spend a task rollout while its required judge is unavailable.
  463. result.update(status="blocked_judge", reason="Web fixed LLM/VLM judge route not configured")
  464. return result
  465. image = smoke.general_base_image()
  466. result.update(base_image_id=image, platform=smoke.GENERAL_PLATFORM, npm_global_prefix="/usr/local", dependency_download_timeout_sec=120)
  467. auth = Path(os.environ.get("CODEX_HOME", str(Path.home() / ".codex"))) / "auth.json"
  468. if not auth.is_file():
  469. return dict(result, status="blocked_auth", reason="Codex auth.json missing")
  470. container = f"oma-wb-{task['subset']}-{os.getpid()}-{int(time.time())}"
  471. result["container"] = container
  472. smoke.execute(["docker", "run", "-d", "--platform", smoke.GENERAL_PLATFORM, "--name", container, "--cpus", "2", "--memory", "6g",
  473. "--mount", f"type=bind,source={runtime},target=/opt/prebuilt,readonly",
  474. "--mount", f"type=bind,source={auth},target=/root/.codex/auth.json,readonly",
  475. "-e", "CODEX_HOME=/root/.codex", "-e", "CI=true", "-e", "OMA_SKIP_VERSION_CHECK=1", "-e", "NPM_CONFIG_PREFIX=/usr/local",
  476. "-e", "PIP_DEFAULT_TIMEOUT=120", "-e", "PIP_RETRIES=5",
  477. "-e", "PYTHONPATH=/workspace:/workspace/src", image, "sleep", "infinity"])
  478. try:
  479. smoke.execute(["docker", "exec", container, "mkdir", "-p", "/workspace"])
  480. seed = frozen_oma_seed(args.cache)
  481. smoke.execute(["docker", "cp", str(seed), f"{container}:/workspace/.agents"])
  482. if task["workspace_archive"]:
  483. smoke.copy_into(container, path / "environment/workspace.tar.gz", "/tmp/workspace.tar.gz")
  484. smoke.copy_into(container, path / "environment/Dockerfile", "/tmp/task-Dockerfile")
  485. scaffold = version_scaffold(task, args.cache)
  486. if scaffold:
  487. smoke.copy_into(container, scaffold["path"], "/tmp/task-pytest-version.py")
  488. result["environment_scaffolds"] = [scaffold]
  489. copy_public_environment(container, path / "environment")
  490. dependency_script = output / "derive-dependencies.py"
  491. dependency_script.write_text(DEPENDENCIES)
  492. smoke.copy_into(container, dependency_script, "/tmp/derive-dependencies.py")
  493. entrypoints_script = output / "install-task-entrypoints.py"
  494. entrypoints_script.write_text(PROJECT_ENTRYPOINTS)
  495. smoke.copy_into(container, entrypoints_script, "/tmp/install-task-entrypoints.py")
  496. compat_script = output / "task-compat-setup.sh"
  497. compat_script.write_text(compatibility_setup((path / "environment/Dockerfile").read_text()))
  498. smoke.copy_into(container, compat_script, "/tmp/task-compat-setup.sh")
  499. fixture_setup = output / "task-postinstall-fixtures.sh"
  500. fixture_command = "python igor.py zip_mods" if "python igor.py zip_mods" in (path / "environment/Dockerfile").read_text() else ""
  501. fixture_setup.write_text("#!/bin/bash\nset -euo pipefail\n" + fixture_command + "\n")
  502. smoke.copy_into(container, fixture_setup, "/tmp/task-postinstall-fixtures.sh")
  503. if fixture_command:
  504. result.setdefault("environment_scaffolds", []).append({"source": "official-Dockerfile-postinstall-fixtures",
  505. "command": fixture_command})
  506. fixture_script = output / "project-fixture-mcp.py"
  507. fixture_script.write_text(MCP_FIXTURE_SETUP)
  508. smoke.copy_into(container, fixture_script, "/tmp/project-fixture-mcp.py")
  509. d_exec(container, BOOTSTRAP, log=output / "setup.log", timeout=600)
  510. result["tooling_python_isolated"] = True
  511. compatibility = d_exec(container, "cat /tmp/task-runtime-compatibility.json")
  512. result["runtime_compatibility"] = json.loads(compatibility.stdout)
  513. metadata = d_exec(container, "cat /tmp/task-project-metadata.json")
  514. result["source_distribution_metadata"] = json.loads(metadata.stdout)
  515. if task["subset"] in {"office", "web"}:
  516. d_exec(container, "python -m pip install --only-binary=:all: openpyxl pandas pypdf python-docx python-pptx lxml beautifulsoup4 Pillow",
  517. log=output / "office-dependencies.log", timeout=300)
  518. if task["subset"] == "web":
  519. d_exec(container, "python -m pip install --only-binary=:all: playwright && python -m playwright install --with-deps chromium",
  520. log=output / "browser-install.log", timeout=600)
  521. if args.mode == "check-env":
  522. result.update(status="environment_ready", score=None, model_invoked=False)
  523. return result
  524. prompt = ("Use the installed OMA skills appropriate to this task and follow workspace instructions. "
  525. "Do not start workflows or spawn additional model processes. Benchmark-local builds and tests are authorized. "
  526. "Do not change or bypass machine build gates. "
  527. "Dependencies are preinstalled; source is importable via PYTHONPATH=/workspace:/workspace/src. "
  528. "Only operate on this benchmark's provided fixture files/services. Do not access real accounts or other network targets.\n\n"
  529. + (path / "instruction.md").read_text())
  530. (output / "prompt.txt").write_text(prompt)
  531. smoke.copy_into(container, output / "prompt.txt", "/tmp/prompt.txt")
  532. if args.saved_agent_patch:
  533. result.update(status="restoring_agent_output", regrade_of=str(args.attempt), model_invoked=False,
  534. saved_agent_patch_sha256=hashlib.sha256(args.saved_agent_patch.read_bytes()).hexdigest())
  535. write_json(output / "status.json", result)
  536. smoke.copy_into(container, args.saved_agent_patch, "/tmp/saved-agent.patch")
  537. d_exec(container, "git apply --binary /tmp/saved-agent.patch", log=output / "restore-agent-output.log")
  538. result["agent_runtime"] = args.original_agent_runtime
  539. else:
  540. result["status"] = "waiting_model"
  541. write_json(output / "status.json", result)
  542. args.model_lock.parent.mkdir(parents=True, exist_ok=True)
  543. with args.model_lock.open("a+") as lock:
  544. fcntl.flock(lock, fcntl.LOCK_EX)
  545. result["status"] = "running_agent"
  546. write_json(output / "status.json", result)
  547. command = '''codex exec -m gpt-6.1-sol -c 'model_reasoning_effort="high"' --dangerously-bypass-approvals-and-sandbox --json --output-last-message /logs/agent/final.md - < /tmp/prompt.txt > /logs/agent/trajectory.jsonl 2> /logs/agent/stderr.log
  548. rc=$?
  549. printf '%s\n' "$rc" > /logs/agent/exit_code
  550. exit "$rc"
  551. '''
  552. agent = d_exec(container, command, log=output / "agent-launch.log",
  553. timeout=min(args.timeout_sec, float(task["timeout_sec"])), check=False)
  554. result["agent_runtime"] = agent_metadata(container)
  555. if agent.returncode:
  556. failure = d_exec(container, "tail -c 3000 /logs/agent/stderr.log", check=False).stdout
  557. failure += json.dumps(result["agent_runtime"].get("failure", {}))
  558. quota = re.search(r"usage limit|quota|rate.limit|limit reached", failure, re.I)
  559. auth_error = re.search(r"unauthorized|authentication|not logged|401", failure, re.I)
  560. result.update(status="waiting_model_quota" if quota else "blocked_auth" if auth_error else "agent_error", reason=f"Agent exit {agent.returncode}")
  561. return result
  562. result["status"] = "running_verifier"
  563. write_json(output / "status.json", result)
  564. env = dict(os.environ, PYTHONPATH=str(upstream / "src"))
  565. argv = [str(venv / "bin/python"), str(Path(__file__).with_name("grade.py")),
  566. "--task", str(path), "--container", container, "--output", str(output)]
  567. if args.judge_config and task["subset"] in {"web", "office"}:
  568. argv += ["--judge-config", str(args.judge_config)]
  569. grade = subprocess.run(argv, env=env, capture_output=True, text=True, timeout=3600)
  570. (output / "verifier-launch.log").write_text(grade.stdout + grade.stderr)
  571. if grade.returncode:
  572. result.update(status="verifier_error", reason=f"Official verifier exit {grade.returncode}")
  573. return result
  574. status, score, detail = result_status(output, task["subset"])
  575. judge_condition = {"code": "official-deterministic-optional-whitebox-disabled",
  576. "office": "native-rules-plus-llm-rubrics" if args.judge_config else "native-rules-only",
  577. "web": "native-fixed-llm-vlm-agent-judges", "sec": "native-programmatic"}[task["subset"]]
  578. result.update(status=status, score=score, verifier=detail,
  579. judge_condition=judge_condition,
  580. judge_timeout_sec=600, office_judge_concurrency=1,
  581. canonical_harbor_run=False, hooks_verified=False)
  582. if args.judge_config and task["subset"] in {"web", "office"}:
  583. route = json.loads(args.judge_config.read_text())
  584. result.update(judge_model=route["model"], judge_effort=route.get("effort", "low"))
  585. return result
  586. except subprocess.TimeoutExpired as exc:
  587. result.update(status="timeout", reason=f"{result['status']}: {exc}")
  588. return result
  589. except Exception as exc:
  590. (output / "exception.log").write_text(traceback.format_exc())
  591. result.update(status="environment_error", reason=f"{type(exc).__name__}: {exc}")
  592. return result
  593. finally:
  594. subprocess.run(["docker", "cp", f"{container}:/logs/agent", str(output)], capture_output=True)
  595. subprocess.run(["docker", "cp", f"{container}:/logs/verifier", str(output / "container-verifier")], capture_output=True)
  596. if not args.keep_containers:
  597. subprocess.run(["docker", "rm", "-f", container], capture_output=True)
  598. result["finished_at"] = time.time()
  599. write_json(output / "status.json", result)
  600. def main():
  601. parser = argparse.ArgumentParser(description=__doc__)
  602. parser.add_argument("mode", choices=["prepare", "inventory", "run", "status", "regrade", "check-env"], nargs="?", default="run")
  603. parser.add_argument("--cache", type=Path, default=Path.home() / ".cache/oma-benchmarks")
  604. parser.add_argument("--output", type=Path, default=Path.home() / ".cache/oma-benchmarks/workbuddy-full")
  605. parser.add_argument("--subsets", nargs="+", choices=list(SUBSETS), default=list(SUBSETS))
  606. parser.add_argument("--task", help="Exact task name (validation only; all tasks remain in inventory)")
  607. parser.add_argument("--attempt", type=Path, help="Completed benign Code attempt containing the saved agent patch for regrade")
  608. parser.add_argument("--max-tasks", type=int)
  609. parser.add_argument("--timeout-sec", type=int, default=1800)
  610. parser.add_argument("--model-lock", type=Path, default=Path.home() / ".cache/oma-benchmarks/model.lock")
  611. parser.add_argument("--judge-config", "--judge-connection", dest="judge_config", type=Path)
  612. parser.add_argument("--allow-build", action="store_true", help="Only use after explicit user authorization for benchmark builds")
  613. parser.add_argument("--build-gate-open", action="store_true", help="Only use while the user's local build-gate GUI approval is active")
  614. parser.add_argument("--retry-errors", action="store_true")
  615. parser.add_argument("--keep-containers", action="store_true")
  616. args = parser.parse_args()
  617. args.saved_agent_patch = None
  618. args.original_agent_runtime = None
  619. args.cache = args.cache.resolve(); args.output = args.output.resolve()
  620. args.output.mkdir(parents=True, exist_ok=True)
  621. if args.mode == "check-env" and not args.task:
  622. parser.error("check-env requires one exact --task and never invokes a model")
  623. if args.mode == "regrade":
  624. if not args.task or not args.attempt:
  625. parser.error("regrade requires --task and --attempt")
  626. original = json.loads((args.attempt / "status.json").read_text())
  627. if original.get("subset") != "code" or original.get("status") in {"preparing", "waiting_model", "running_agent", "running_verifier"}:
  628. parser.error("regrade requires a completed benign Code attempt")
  629. if original.get("task") != args.task:
  630. parser.error("regrade task must match the saved attempt")
  631. exit_code = args.attempt / "agent/exit_code"
  632. if not exit_code.is_file() or exit_code.read_text().strip() != "0":
  633. parser.error("regrade requires a saved successful agent exit")
  634. patch = next((args.attempt / name for name in ["verifier/agent.patch", "container-verifier/agent.patch"]
  635. if (args.attempt / name).is_file()), None)
  636. if patch is None:
  637. parser.error("No saved agent.patch in the selected attempt")
  638. saved = args.output / "regrade-inputs" / f"{time.time_ns()}.patch"
  639. saved.parent.mkdir(parents=True, exist_ok=True);saved.write_bytes(patch.read_bytes())
  640. args.saved_agent_patch = saved
  641. args.original_agent_runtime = original.get("agent_runtime")
  642. args.retry_errors = True
  643. manifest_path = args.output / "manifest.json"
  644. if args.mode == "status" and manifest_path.exists():
  645. manifest = json.loads(manifest_path.read_text())
  646. for task in manifest["tasks"]:
  647. status_path = args.output / task["subset"] / task["task"] / "status.json"
  648. if status_path.exists():
  649. task.update(json.loads(status_path.read_text()))
  650. else:
  651. upstream, runtime, venv = prepare(args)
  652. existing = {item["id"]: item for item in json.loads(manifest_path.read_text())["tasks"]} if manifest_path.exists() else {}
  653. tasks = [existing.get(item["id"], item) for item in inventory(upstream)]
  654. manifest = dict(benchmark="WorkBuddy Bench v1.0", expected_tasks=260,
  655. upstream_revision=smoke.REVISION, dataset_sha256=CHECKSUMS,
  656. subsets=SUBSETS, tasks=tasks)
  657. write_json(manifest_path, manifest)
  658. if args.mode in {"status", "inventory", "prepare"}:
  659. print(json.dumps(collections.Counter(item["status"] for item in manifest["tasks"]), indent=2))
  660. return
  661. processed = 0
  662. for task in manifest["tasks"]:
  663. if (args.output / "STOP_AFTER_TASK").exists():
  664. print("Stopped at task boundary: STOP_AFTER_TASK exists", flush=True)
  665. break
  666. if task["subset"] not in args.subsets or (args.task and task["task"] != args.task):
  667. continue
  668. task_status_path = args.output / task["subset"] / task["task"] / "status.json"
  669. if task_status_path.exists():
  670. task.update(json.loads(task_status_path.read_text()))
  671. if args.mode != "regrade" and (task["status"] == "scored" or (task["status"] not in {"pending", "preparing", "waiting_model", "running_agent", "running_verifier"} and not args.retry_errors)):
  672. continue
  673. print(f"RUN {task['id']}", flush=True)
  674. try:
  675. result = run_task(task, args, upstream, runtime, venv)
  676. except Exception as exc:
  677. result = dict(task, status="environment_error", score=None, reason=str(exc))
  678. task.clear()
  679. task.update(result)
  680. task.setdefault("finished_at", time.time())
  681. write_json(task_status_path, task)
  682. manifest["updated_at"] = time.time()
  683. manifest["counts"] = dict(collections.Counter(item["status"] for item in manifest["tasks"]))
  684. write_json(manifest_path, manifest)
  685. print(f"RESULT {task['id']} {task['status']} score={task.get('score')}", flush=True)
  686. if task["status"] in {"waiting_model_quota", "blocked_auth"}:
  687. print("Batch stopped: model authentication or quota requires recovery", flush=True)
  688. break
  689. processed += 1
  690. if args.max_tasks and processed >= args.max_tasks:
  691. break
  692. if __name__ == "__main__":
  693. main()

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.