Skip to content
Back to search

webcmd

Self-learning agent browser

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/agentrhq/webcmd with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/agentrhq/webcmd'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/agentrhq--webcmd/badge.svg)](/listings/agentrhq--webcmd)

HTML

<a href="/listings/agentrhq--webcmd"><img src="/api/listings/agentrhq--webcmd/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Agent prompt

Fetch and follow https://raw.githubusercontent.com/agentrhq/webcmd/main/start.md to set up Webcmd end to end.

Manual

Webcmd requires Node.js 20.6+.

npm install -g @agentrhq/webcmd
webcmd skills add

When prompted, choose Claude, Codex, another supported harness, or a custom skills path. That installs exactly one skill, `webcmd-browser`.

Load or tag `webcmd-browser` only for live browser work, then describe the outcome you want. Installation and setup commands do not require that skill.

Use webcmd to research the latest discussions about browser automation across Hacker News and Reddit, then return a concise comparison with source links.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 17 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "$schema": "https://www.schemastore.org/claude-code-marketplace.json",
  3. "name": "webcmd",
  4. "description": "Self-learning browser infrastructure for AI agents: live browser control plus local sitemap memory that makes repeat runs faster and cheaper.",
  5. "owner": {
  6. "name": "AgentRHQ",
  7. "url": "https://github.com/agentrhq"
  8. },
  9. "plugins": [
  10. {
  11. "name": "webcmd",
  12. "description": "Webcmd browser skill for live, authenticated browser automation with local sitemap memory. Requires the @agentrhq/webcmd npm CLI.",
  13. "source": "./",
  14. "category": "productivity"
  15. }
  16. ]
  17. }
README.md · README · 166 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <img width="1280" height="640" alt="Webcmd — stop paying agents to rediscover the web" src="docs/readme-hero-v2.png" />
  2. <p align="center">
  3. <a href="https://www.npmjs.com/package/@agentrhq/webcmd">
  4. <img alt="NPM version" src="https://img.shields.io/npm/v/@agentrhq/webcmd.svg?style=for-the-badge&color=1E88E5&labelColor=000000">
  5. </a>
  6. <a href="https://webcmd.dev/docs">
  7. <img alt="Documentation" src="https://img.shields.io/badge/docs-webcmd.dev-7C3AED.svg?style=for-the-badge&labelColor=000000">
  8. </a>
  9. <a href="https://github.com/agentrhq/webcmd/blob/main/LICENSE">
  10. <img alt="License" src="https://img.shields.io/badge/license-Apache%202.0-1E88E5.svg?style=for-the-badge&labelColor=000000">
  11. </a>
  12. <a href="https://discord.gg/9YP2C9tvMp">
  13. <img alt="Join the community on Discord" src="https://img.shields.io/badge/Join%20the%20community-5865F2.svg?style=for-the-badge&logo=discord&logoColor=white&labelColor=000000&logoWidth=20">
  14. </a>
  15. <a href="https://x.com/agentrhq">
  16. <img alt="Follow AgentR on X" src="https://img.shields.io/badge/Built%20by%20%40agentrhq-000000.svg?style=for-the-badge&logo=x&logoColor=white&labelColor=000000&logoWidth=20">
  17. </a>
  18. </p>
  19. # Webcmd
  20. **Self-learning browser infra for AI agents.**
  21. Webcmd learns the navigational context of websites as agents use them, then
  22. turns that knowledge into local memory for faster, cheaper, more reliable
  23. browser automation. The goal is simple: stop making agents rediscover the same
  24. sites on every run and cut browser-agent token spend by up to 90%.
  25. Webcmd pairs live browser control with a self-learning memory layer:
  26. | Layer | Scenario | What Webcmd Helps With |
  27. | --- | --- | --- |
  28. | 0. Live browser control | The site is unfamiliar. | Use `webcmd browser` to inspect, click, type, extract, capture network calls, and complete the task in a real browser. |
  29. | 1. Sitemap memory | The site is familiar, but the action space is not fully known. | Capture an agent-facing sitemap of observed pages, states, actions, workflows, APIs, pitfalls, and fallback paths. |
  30. ## Demo
  31. https://github.com/user-attachments/assets/bdb65307-9e2a-4d58-9175-45d59528ae37
  32. ## Proxy Partner Giveaway (courtesy: Swiftproxy)
  33. Power your web automation and data collection with Swiftproxy, a residential proxy provider offering rotating and static residential proxies for reliable web access. With 90M+ residential IPs across 220+ locations, Swiftproxy helps support automated browsing and web-based workflows. Get 10% off with code PROXY90.
  34. Link: https://www.swiftproxy.net/?ref=webcmd
  35. ## Quick Start
  36. ### Agent prompt
  37. ```text
  38. Fetch and follow https://raw.githubusercontent.com/agentrhq/webcmd/main/start.md to set up Webcmd end to end.
  39. ```
  40. ### Manual
  41. Webcmd requires Node.js 20.6+.
  42. ```bash
  43. npm install -g @agentrhq/webcmd
  44. webcmd skills add
  45. ```
  46. When prompted, choose Claude, Codex, another supported harness, or a custom
  47. skills path. That installs exactly one skill, `webcmd-browser`.
  48. Load or tag `webcmd-browser` only for live browser work, then describe the
  49. outcome you want. Installation and setup commands do not require that skill.
  50. ```text
  51. Use webcmd to research the latest discussions about browser automation across Hacker News and Reddit, then return a concise comparison with source links.
  52. ```
  53. ## What You Can Ask
  54. - “Use webcmd to research agentic browser automation on PubMed and return the title, authors, publication date, abstract, and URL for each result.”
  55. - “Use webcmd to find active AI infrastructure companies in the YC company directory and return the company, batch, description, location, profile URL, and source links. Keep it read-only.”
  56. - “Use webcmd to look up parts on Grainger by part number and return price, stock, minimum order quantity, lead time, and product URL.”
  57. - “Use webcmd with my logged-in `work` profile to summarize unread LinkedIn messages from the last seven days and return the sender, subject or opening text, received time, and conversation URL.”
  58. - “Use webcmd to check Grainger part prices and SAP Ariba purchase-order status, then return a combined summary.”
  59. ## See It in Action
  60. ```text
  61. Use webcmd with my logged-in `social` profile to collect my recent X bookmarks and return the author, text, and URL.
  62. ```
  63. The agent uses the logged-in profile to complete the task in a real browser.
  64. Along the way, Webcmd quietly retains useful navigation context so later agents
  65. can avoid repeating the same exploration.
  66. ## Where Webcmd Works
  67. Webcmd can work through authenticated browser sessions across research, social,
  68. AI, shopping, and booking products.
  69. | Group | Supported surfaces | Representative outcomes |
  70. | --- | --- | --- |
  71. | research and communities | Hacker News, Reddit, PubMed | Compare current discussions, find primary research, and return concise summaries with source links. |
  72. | social and professional | X/Twitter, LinkedIn, TikTok | Collect bookmarks, monitor public posts, or research people and creators with a named profile when needed. |
  73. | AI tools | ChatGPT, Claude, Gemini, NotebookLM | Retrieve conversations, research outputs, notebooks, and generated materials from the tools you already use. |
  74. | shopping and bookings | Amazon, Blinkit, Zepto, BigBasket, District, Practo | Compare products, availability, prices, appointments, events, and delivery options. |
  75. This list is illustrative. Webcmd can operate other websites through the same
  76. live browser workflow.
  77. ## How Self-Learning Works
  78. <img width="1672" height="941" alt="How Webcmd learns: load memory, use the live web, keep useful learnings, and help the next agent" src="docs/readme-self-learning.png" />
  79. Learning stays quiet and selective: the live browser is always truth, Webcmd
  80. never explores just to learn, and a memory failure never blocks the task. First
  81. access may use a Webcmd Cloud seed; subsequent learning stays local.
  82. For local, multi-step browser exploration, agents can send one sandboxed
  83. Playwright-style program to an explicit browser session:
  84. ```bash
  85. webcmd --profile work session create "Work Project" -f json
  86. # id: work-project-k7
  87. webcmd --profile work --session work-project-k7 browser tabs
  88. webcmd --profile work --session work-project-k7 browser run --file explore.js
  89. printf 'return await page.title();' \
  90. | webcmd --profile work --session work-project-k7 browser run --stdin
  91. webcmd --profile work session close work-project-k7
  92. ```
  93. Profiles are cookie jars; Sessions are independent browser windows within a
  94. profile, so Session IDs are immutable, Profile-scoped, and safe to reuse for
  95. that Session's lifetime. Parallel agents should create separate Sessions.
  96. Raw browser commands require an explicit readable Session ID.
  97. ## Benchmarks
  98. On [BU Bench V1](https://github.com/browser-use/benchmark#bu-bench-v1), a
  99. 100-task browser automation benchmark, Webcmd recorded the highest accuracy and
  100. lowest estimated controller cost per completed task, and fewest agent turns per
  101. completed task in this comparison.
  102. ![BU Bench V1 comparison: webcmd leads accuracy at 67%, cost per completed task at $0.255, and agent turns per completed task at 9.8](./benchmarks/charts/bu-bench-readme.svg)
  103. All tools used the same Pi controller, controller model, Codex `gpt-5.4` judge,
  104. and CloakBrowser engine. This is a stronger judge than the original BU Bench
  105. setup, whose [current runner uses Gemini 2.5 Flash](https://github.com/browser-use/benchmark/blob/main/run_eval.py#L37-L38).
  106. Accuracy is passed tasks out of 100. Cost and agent turns are averaged over
  107. completed tasks; cost excludes judge usage. See the
  108. [benchmark report](./benchmarks/README.md) for category results, methodology,
  109. architectural analysis, and reproduction steps.
  110. ## Learn More
  111. Webcmd Cloud can run supported commands and browser sessions on hosted infrastructure. It is in active development and is not yet stable.
  112. - [Prompt Cookbook](https://webcmd.dev/docs/agent-prompts)
  113. - [How Webcmd Works](https://webcmd.dev/docs/concepts)
  114. - [Local or Cloud](https://webcmd.dev/docs/local-or-cloud)
  115. - [Command Surface](https://webcmd.dev/docs/cli-reference)
  116. ## Contributing
  117. See [CONTRIBUTING.md](./CONTRIBUTING.md).
  118. ## License
  119. Released under the terms in [`LICENSE`](./LICENSE).
benchmarks/scripts/run_controller.py · Bundled script · 1,699 lines · 1 finding marked
  1. #!/usr/bin/env python3
  2. import asyncio
  3. import base64
  4. import hashlib
  5. import json
  6. import math
  7. import os
  8. import re
  9. import shlex
  10. import shutil
  11. import signal
  12. import subprocess
  13. import threading
  14. import time
  15. from dataclasses import dataclass
  16. from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
  17. from pathlib import Path
  18. from typing import Literal
  19. from urllib.parse import unquote
  20. from agent_browser_runtime import start_agent_browser_runtime
  21. from axi_runtime import start_axi_runtime
  22. from browser_use_runtime import start_browser_use_runtime
  23. from dev_browser_runtime import start_dev_browser_runtime
  24. from libretto_runtime import start_libretto_runtime
  25. from playwright_cli_runtime import start_playwright_cli_runtime
  26. Controller = Literal["codex", "claude", "pi"]
  27. Tool = Literal["webcmd", "chrome-devtools-axi", "agent-browser", "dev-browser", "libretto", "browser-use", "playwright-cli"]
  28. Termination = Literal["completed", "timeout", "controller_error", "missing_final_answer", "tool_policy_violation"]
  29. FINAL_ANSWER_RE = re.compile(r"^FINAL ANSWER:[ \t]*(\S[^\r\n]*)$", re.MULTILINE)
  30. ESSENTIAL_ENV_KEYS = {
  31. "HOME", "LANG", "LANGUAGE", "LC_ALL", "LC_CTYPE", "LOGNAME", "NODE_EXTRA_CA_CERTS",
  32. "PATH", "SHELL", "SSL_CERT_DIR", "SSL_CERT_FILE", "TEMP", "TERM", "TMP", "TMPDIR", "USER",
  33. "XDG_CACHE_HOME", "XDG_CONFIG_HOME", "XDG_DATA_HOME", "XDG_STATE_HOME",
  34. }
  35. CONTROLLER_ENV_PREFIXES = {
  36. "codex": ("CODEX_", "OPENAI_"),
  37. "claude": ("ANTHROPIC_", "CLAUDE_"),
  38. "pi": ("OPENAI_", "ANTHROPIC_", "PI_"),
  39. }
  40. CONTROLLER_ENV_KEYS = {
  41. "codex": set(),
  42. "claude": {
  43. "AWS_ACCESS_KEY_ID", "AWS_DEFAULT_REGION", "AWS_PROFILE", "AWS_REGION", "AWS_SECRET_ACCESS_KEY",
  44. "AWS_SESSION_TOKEN", "CLOUD_ML_REGION", "GOOGLE_APPLICATION_CREDENTIALS",
  45. },
  46. "pi": set(),
  47. }
  48. TOOL_ENV_PREFIXES = {
  49. "webcmd": ("WEBCMD_",),
  50. "chrome-devtools-axi": ("CHROME_DEVTOOLS_AXI_", "CHROME_DEVTOOLS_MCP_"),
  51. "agent-browser": (),
  52. "dev-browser": (),
  53. "libretto": (),
  54. "browser-use": (),
  55. "playwright-cli": (),
  56. }
  57. EVALUATION_ENV_KEYS = {
  58. "DATASET_DECRYPTION_KEY", "DATASET_PATH", "EVIDENCE_PATH", "EXPECTED_ANSWER", "GOOGLE_API_KEY",
  59. "GROUND_TRUTH", "JUDGE_MODEL", "JUDGE_PROMPT", "PRIOR_VERDICT", "RESULTS_DIR",
  60. }
  61. EVALUATION_ENV_MARKERS = (
  62. "ANSWER", "ATTEMPT", "DATASET", "DECRYPT", "EVIDENCE", "GROUND_TRUTH", "GROUNDTRUTH",
  63. "JUDGE", "PROMPT", "RESULT", "VERDICT",
  64. )
  65. CLEANUP_TIMEOUT_SECONDS = 5
  66. WEBCMD_BENCHMARK_PROFILE = "benchmark"
  67. WEBCMD_SESSION_RE = re.compile(
  68. r"^session_[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$",
  69. re.IGNORECASE,
  70. )
  71. PI_CONTROLLER = Path(__file__).resolve().with_name("pi_controller.mjs")
  72. LIBRETTO_MCP = Path(__file__).resolve().with_name("libretto_mcp.mjs")
  73. LIBRETTO_TOOLS = (
  74. "browser_open",
  75. "browser_exec",
  76. "browser_snapshot",
  77. "browser_status",
  78. "browser_close",
  79. )
  80. WEBCMD_BROWSER_SKILL = Path.home() / ".codex/skills/webcmd-browser"
  81. WEBCMD_BROWSER_SKILL_FILE = WEBCMD_BROWSER_SKILL / "SKILL.md"
  82. WEBCMD_BROWSER_SKILL_ROOT = WEBCMD_BROWSER_SKILL.resolve()
  83. DEV_BROWSER_SKILL = Path.home() / ".codex/skills/dev-browser"
  84. BROWSER_USE_SKILL = Path.home() / ".codex/skills/browser-use"
  85. AGENT_BROWSER_SKILL = Path.home() / ".codex/skills/agent-browser"
  86. PLAYWRIGHT_CLI_SKILL = Path.home() / ".codex/skills/playwright-cli"
  87. PI_SETUP_SKILL_FILES = {
  88. "webcmd": frozenset({WEBCMD_BROWSER_SKILL_FILE.resolve()}),
  89. "dev-browser": frozenset({(DEV_BROWSER_SKILL / "SKILL.md").resolve()}),
  90. "browser-use": frozenset({(BROWSER_USE_SKILL / "SKILL.md").resolve()}),
  91. "agent-browser": frozenset({(AGENT_BROWSER_SKILL / "SKILL.md").resolve()}),
  92. "playwright-cli": frozenset({(PLAYWRIGHT_CLI_SKILL / "SKILL.md").resolve()}),
  93. }
  94. GPT_5_6_SOL_PRICES_PER_MILLION = {
  95. "input": 5.0,
  96. "cached_input": 0.5,
  97. "cache_write_input": 6.25,
  98. "output": 30.0,
  99. }
  100. GPT_5_6_SOL_MODELS = frozenset({"gpt-5.6", "gpt-5.6-sol"})
  101. LONG_CONTEXT_INPUT_THRESHOLD = 272_000
  102. class _CodexTurnCollector:
  103. def __init__(self) -> None:
  104. self._payloads: list[dict] = []
  105. self._lock = threading.Lock()
  106. self._server: ThreadingHTTPServer | None = None
  107. self._thread: threading.Thread | None = None
  108. def __enter__(self) -> "_CodexTurnCollector":
  109. collector = self
  110. class Handler(BaseHTTPRequestHandler):
  111. def do_POST(self) -> None:
  112. length = int(self.headers.get("Content-Length", "0"))
  113. try:
  114. payload = json.loads(self.rfile.read(length))
  115. except (json.JSONDecodeError, UnicodeDecodeError):
  116. self.send_error(400)
  117. return
  118. if isinstance(payload, dict):
  119. with collector._lock:
  120. collector._payloads.append(payload)
  121. body = b"{}"
  122. self.send_response(200)
  123. self.send_header("Content-Type", "application/json")
  124. self.send_header("Content-Length", str(len(body)))
  125. self.end_headers()
  126. self.wfile.write(body)
  127. def log_message(self, format: str, *args: object) -> None:
  128. pass
  129. self._server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
  130. self._server.daemon_threads = True
  131. self._thread = threading.Thread(target=self._server.serve_forever, daemon=True)
  132. self._thread.start()
  133. return self
  134. def __exit__(self, *args: object) -> None:
  135. if self._server is not None:
  136. self._server.shutdown()
  137. self._server.server_close()
  138. if self._thread is not None:
  139. self._thread.join()
  140. @property
  141. def endpoint(self) -> str:
  142. if self._server is None:
  143. raise RuntimeError("Codex turn collector is not running")
  144. host, port = self._server.server_address
  145. return f"http://{host}:{port}/v1/logs"
  146. @property
  147. def agent_turns(self) -> int | None:
  148. with self._lock:
  149. payloads = list(self._payloads)
  150. if not payloads:
  151. return None
  152. return sum(
  153. 1
  154. for payload in payloads
  155. for resource in payload.get("resourceLogs", [])
  156. for scope in resource.get("scopeLogs", [])
  157. for record in scope.get("logRecords", [])
  158. if self._is_completed_model_response(record)
  159. )
  160. @staticmethod
  161. def _is_completed_model_response(record: object) -> bool:
  162. if not isinstance(record, dict):
  163. return False
  164. attributes = {
  165. attribute.get("key"): attribute.get("value")
  166. for attribute in record.get("attributes", [])
  167. if isinstance(attribute, dict)
  168. }
  169. def value(key: str) -> object:
  170. wrapped = attributes.get(key)
  171. if not isinstance(wrapped, dict):
  172. return None
  173. return next(iter(wrapped.values()), None)
  174. return (
  175. value("event.name") == "codex.sse_event"
  176. and value("event.kind") == "response.completed"
  177. and "input_token_count" in attributes
  178. and "error.message" not in attributes
  179. )
  180. @dataclass
  181. class ExecutionEvidence:
  182. final_answer: str
  183. steps: list[str]
  184. screenshot_paths: list[Path]
  185. controller_exit_code: int
  186. termination: Termination
  187. metrics: "ControllerMetrics | None" = None
  188. @dataclass(frozen=True)
  189. class TokenUsage:
  190. input: int
  191. cache_read_input: int
  192. cache_creation_input: int
  193. non_cached_input: int
  194. output: int
  195. reasoning_output: int | None
  196. total: int
  197. estimated_api_cost_usd: float | None = None
  198. @dataclass(frozen=True)
  199. class ControllerMetrics:
  200. duration_seconds: float
  201. steps: int
  202. tool_calls: int
  203. tokens: TokenUsage | None
  204. provider_turns: int | None
  205. provider_duration_seconds: float | None
  206. provider_api_duration_seconds: float | None
  207. agent_turns: int | None = None
  208. @dataclass
  209. class ParsedEvents:
  210. steps: list[str]
  211. commands: list[str]
  212. event_types: list[str]
  213. final_text: str
  214. steps_count: int
  215. tool_calls: int
  216. tokens: TokenUsage | None
  217. provider_turns: int | None
  218. provider_duration_seconds: float | None
  219. provider_api_duration_seconds: float | None
  220. agent_turns: int | None
  221. mcp_calls: list[tuple[str, str]]
  222. screenshot_images: list[bytes]
  223. def _build_prompt(tool: Tool, session: str, shots_dir: Path, task: str) -> str:
  224. first_shot = shots_dir / "step_001.png"
  225. if tool == "chrome-devtools-axi":
  226. tool_rules = """- Use only `npx -y chrome-devtools-axi` for browser interaction.
  227. - Every shell command must begin with exactly `npx -y chrome-devtools-axi`; use no substitutions, redirections, pipes, or other executables.
  228. - Do not use Web search, browser MCPs, Playwright, Puppeteer, curl, wget, raw HTTP, or any non-AXI automation tool.
  229. - Use the `$chrome-devtools-axi` skill for AXI usage guidance.
  230. - The AXI session and its dedicated CloakBrowser connection are already configured in the environment; do not start another browser."""
  231. elif tool == "agent-browser":
  232. tool_rules = """- Use only `agent-browser` for browser interaction.
  233. - Use one `agent-browser` command per shell invocation; use no substitutions, redirections, pipes, or other executables.
  234. - Do not use `batch`, `close`, connection/profile flags, or multiple agent-browser commands in one shell invocation.
  235. - Never pass a URL to `agent-browser read`; use `agent-browser open URL` and then `agent-browser read` so all page traffic stays inside CloakBrowser.
  236. - Do not use Web search, browser MCPs, Playwright, Puppeteer, curl, wget, raw HTTP, or any non-agent-browser automation tool.
  237. - Use the `$agent-browser` skill for agent-browser usage guidance. Load current CLI instructions with `agent-browser skills get core` before the first browser command.
  238. - The agent-browser session and its dedicated CloakBrowser connection are already configured in the environment; do not start, connect, configure, or close another browser."""
  239. elif tool == "playwright-cli":
  240. tool_rules = """- Use only `playwright-cli` for browser interaction.
  241. - Use one `playwright-cli` command per shell invocation; use no substitutions, redirections, pipes, or other executables.
  242. - Do not use `attach`, `detach`, `close`, `close-all`, `kill-all`, `install`, `install-browser`, `show`, connection/profile/browser flags, or multiple playwright-cli commands in one shell invocation.
  243. - Prefer `playwright-cli goto URL` and `playwright-cli snapshot` on the already attached session. Do not start Playwright's own Chromium.
  244. - Do not use Web search, browser MCPs, Puppeteer, curl, wget, raw HTTP, or any non-playwright-cli automation tool.
  245. - Use the `$playwright-cli` skill for playwright-cli usage guidance.
  246. - The playwright-cli session and its dedicated CloakBrowser connection are already configured in the environment; do not start, connect, configure, or close another browser."""
  247. elif tool == "dev-browser":
  248. tool_rules = f"""- Use only `dev-browser` for browser interaction.
  249. - Invoke `dev-browser` with one quoted heredoc per shell command. The heredoc body must contain only the sandboxed JavaScript described by the `$dev-browser` skill.
  250. - You may add only `--timeout SECONDS`; do not use `--browser`, `--connect`, `--headless`, `run`, `stop`, or any other subcommand or connection option.
  251. - Use no substitutions, pipes, extra redirections, or other executables.
  252. - Do not use Web search, browser MCPs, external Playwright, Puppeteer, curl, wget, raw HTTP, or any non-dev-browser automation tool.
  253. - Use the `$dev-browser` skill for dev-browser usage guidance.
  254. - The dev-browser command is already pinned to this task's dedicated CloakBrowser connection; do not start, connect, configure, or close another browser.
  255. - Save screenshots with `await saveScreenshot(await page.screenshot(), "{session}-step_001.png")`, then `{session}-step_002.png`, and so on."""
  256. elif tool == "libretto":
  257. tool_rules = """- Use only the Libretto MCP tools `browser_open`, `browser_exec`, `browser_snapshot`, `browser_status`, and `browser_close` for browser interaction.
  258. - Do not use shell commands, Web Search, other MCP servers or tools, Playwright outside `browser_exec`, Puppeteer, curl, wget, or raw HTTP.
  259. - The Libretto provider is already pinned to this task's dedicated CloakBrowser; open it with `browser_open` and do not configure another browser.
  260. - Reuse the session ID returned by `browser_open` and close it with `browser_close` when the task is complete."""
  261. elif tool == "browser-use":
  262. tool_rules = f"""- Use only `browser-use` for browser interaction.
  263. - Invoke `browser-use` with one quoted heredoc per shell command. The heredoc body must contain only the Python helpers described by the `$browser-use` skill.
  264. - Do not pass `--cdp-url`, `--connect`, `--profile`, `--headed`, `auth`, `--doctor`, `--update`, `mac-approve`, `cloud`, or any other connection or cloud option.
  265. - Do not prefix the command with environment assignments. Do not call `start_remote_daemon`, `stop_remote_daemon`, or `auth login`.
  266. - Use no substitutions, pipes, extra redirections, or other executables.
  267. - Do not use Web search, browser MCPs, Playwright, Puppeteer, curl, wget, raw HTTP, or any non-browser-use automation tool.
  268. - Use the `$browser-use` skill for browser-use usage guidance.
  269. - The browser-use command is already pinned to this task's dedicated CloakBrowser connection; do not start, connect, configure, or close another browser.
  270. - Save screenshots with `capture_screenshot("{first_shot}")`, then `step_002.png`, and so on in the same directory."""
  271. else:
  272. tool_rules = f"""- Use only `webcmd` raw-browser commands for task execution. Do not use Web Search, browser MCPs, external Playwright, Puppeteer, curl, wget, raw HTTP, adapters, fetch commands, plugins, or any non-Webcmd automation tool.
  273. - Use the `$webcmd-browser` skill for browser guidance. Do not load any other Webcmd skill. This benchmark's raw-browser route is already selected. Skip adapter and plugin discovery, the top-level Webcmd router, `webcmd doctor`, and the skill's Session lifecycle steps.
  274. - The harness already created the dedicated Profile and Session. Use `--profile {WEBCMD_BENCHMARK_PROFILE}` and `--session {session}` on every browser command.
  275. - Do not create, close, or force-close Sessions, manage the daemon, or change Profiles. The harness owns Session cleanup after the controller exits.
  276. - Use `browser run --stdin`, not `run --file`. Other than reading the Webcmd browser skill and its references, every shell command must execute only `webcmd`; do not use shell helpers, substitutions, pipes, or other executables."""
  277. if tool == "dev-browser":
  278. screenshot_rules = f"""- Save screenshots after meaningful page transitions and at the final state with `saveScreenshot`.
  279. - Name screenshots `{session}-step_001.png`, then `{session}-step_002.png`, `{session}-step_003.png`, and so on without overwriting. The harness collects them automatically."""
  280. elif tool == "browser-use":
  281. screenshot_rules = f"""- Save screenshots after meaningful page transitions and at the final state with `capture_screenshot`.
  282. - Name screenshots `{first_shot}`, then step_002.png, step_003.png, and so on without overwriting."""
  283. elif tool == "libretto":
  284. screenshot_rules = """- Call `browser_snapshot` with `screenshot: true` after meaningful page transitions and at the final state.
  285. - The harness collects those native Libretto screenshot results automatically."""
  286. elif tool == "webcmd":
  287. screenshot_rules = """- Save screenshots after meaningful page transitions and at the final state.
  288. - In Webcmd run programs, screenshot paths must be relative logical filenames such as `shots/step_001.png`, then `shots/step_002.png`, without `..` or absolute paths."""
  289. else:
  290. screenshot_rules = f"""- Save screenshots after meaningful page transitions and at the final state.
  291. - Name screenshots `{first_shot}`, then step_002.png, step_003.png, and so on without overwriting."""
  292. return f"""You are executing a browser benchmark task.
  293. Hard rules:
  294. {tool_rules}
  295. - Quote URLs that contain `?`, `&`, `#`, `(`, `)`, or other shell punctuation.
  296. - Do not change filters, sort modes, chart variants, regions, or page views unless the task explicitly asks.
  297. {screenshot_rules}
  298. - If a CAPTCHA, human-verification control, or anti-bot challenge is interactable, make a reasonable in-browser attempt to complete it and continue. If a login or authentication page blocks further progress, report that as the outcome. Otherwise, only report failure when anti-bot protection remains blocking.
  299. - Work autonomously and do not ask clarifying questions.
  300. - End with exactly one line `FINAL ANSWER: answer` and nothing after it. Use `FINAL ANSWER: done` for action-only tasks.
  301. Task:
  302. {task}
  303. """
  304. def _controller_command(
  305. controller: Controller,
  306. model: str,
  307. prompt: str,
  308. reasoning_effort: str | None = None,
  309. tool: Tool | None = None,
  310. runtime_env: dict[str, str] | None = None,
  311. otel_endpoint: str | None = None,
  312. ) -> tuple[list[str], bytes | None]:
  313. if controller == "codex":
  314. command = ["codex", "exec", "--json", "--model", model]
  315. if reasoning_effort is not None:
  316. command.extend(["-c", f'model_reasoning_effort="{reasoning_effort}"'])
  317. if otel_endpoint is not None:
  318. command.extend(
  319. [
  320. "-c",
  321. "otel.exporter={otlp-http={"
  322. f"endpoint={json.dumps(otel_endpoint)},protocol=\"json\"}}}}",
  323. ]
  324. )
  325. if tool == "libretto":
  326. cdp_url = (runtime_env or {}).get("LIBRETTO_CDP_URL")
  327. if not cdp_url:
  328. raise ValueError(
  329. "Libretto requires a task-private LIBRETTO_CDP_URL"
  330. )
  331. enabled_tools = ",".join(json.dumps(name) for name in LIBRETTO_TOOLS)
  332. command.extend(
  333. [
  334. "-c",
  335. 'mcp_servers.libretto.command="node"',
  336. "-c",
  337. f"mcp_servers.libretto.args=[{json.dumps(str(LIBRETTO_MCP))}]",
  338. "-c",
  339. "mcp_servers.libretto.required=true",
  340. "-c",
  341. f"mcp_servers.libretto.enabled_tools=[{enabled_tools}]",
  342. "-c",
  343. "mcp_servers.libretto.env="
  344. f'{{LIBRETTO_CDP_URL={json.dumps(cdp_url)}}}',
  345. ]
  346. )
  347. command.extend(["--sandbox", "danger-full-access", "--skip-git-repo-check", "--ignore-user-config", "-"])
  348. return command, prompt.encode()
  349. if controller == "claude":
  350. return (["claude", "-p", prompt, "--model", model, "--dangerously-skip-permissions", "--output-format", "stream-json", "--verbose", "--no-session-persistence", "--bare"], None)
  351. if controller == "pi":
  352. command = [
  353. "node",
  354. str(PI_CONTROLLER),
  355. "--model",
  356. model,
  357. ]
  358. pi_tool = tool or "webcmd"
  359. if pi_tool == "webcmd":
  360. command.extend(["--skill-path", str(WEBCMD_BROWSER_SKILL)])
  361. elif pi_tool == "dev-browser":
  362. command.extend(
  363. ["--tool", pi_tool, "--skill-path", str(DEV_BROWSER_SKILL)]
  364. )
  365. elif pi_tool == "agent-browser":
  366. if not (runtime_env or {}).get("AGENT_BROWSER_CDP"):
  367. raise ValueError(
  368. "Pi agent-browser requires a task-private AGENT_BROWSER_CDP"
  369. )
  370. command.extend(
  371. ["--tool", pi_tool, "--skill-path", str(AGENT_BROWSER_SKILL)]
  372. )
  373. elif pi_tool == "playwright-cli":
  374. if not (runtime_env or {}).get("PLAYWRIGHT_CLI_SESSION"):
  375. raise ValueError(
  376. "Pi playwright-cli requires a task-private PLAYWRIGHT_CLI_SESSION"
  377. )
  378. command.extend(
  379. ["--tool", pi_tool, "--skill-path", str(PLAYWRIGHT_CLI_SKILL)]
  380. )
  381. elif pi_tool == "browser-use":
  382. if not (runtime_env or {}).get("BU_CDP_URL"):
  383. raise ValueError(
  384. "Pi browser-use requires a task-private BU_CDP_URL"
  385. )
  386. command.extend(
  387. ["--tool", pi_tool, "--skill-path", str(BROWSER_USE_SKILL)]
  388. )
  389. elif pi_tool == "libretto":
  390. if not (runtime_env or {}).get("LIBRETTO_CDP_URL"):
  391. raise ValueError(
  392. "Pi Libretto requires a task-private LIBRETTO_CDP_URL"
  393. )
  394. command.extend(["--tool", pi_tool])
  395. else:
  396. raise ValueError(f"Pi support is not configured for {pi_tool}")
  397. if reasoning_effort is not None:
  398. command.extend(["--thinking", reasoning_effort])
  399. return command, prompt.encode()
  400. raise ValueError(f"unknown controller: {controller}")
  401. def _subprocess_env(controller: Controller | None = None, tool: Tool | None = None) -> dict[str, str]:
  402. allowed_keys = ESSENTIAL_ENV_KEYS | (CONTROLLER_ENV_KEYS[controller] if controller else set())
  403. allowed_prefixes = (CONTROLLER_ENV_PREFIXES[controller] if controller else ()) + (TOOL_ENV_PREFIXES[tool] if tool else ())
  404. env = {
  405. key: value
  406. for key, value in os.environ.items()
  407. if key not in EVALUATION_ENV_KEYS
  408. and not any(marker in key.upper() for marker in EVALUATION_ENV_MARKERS)
  409. and (key in allowed_keys or key.startswith(allowed_prefixes))
  410. }
  411. if tool == "webcmd":
  412. node_bin = Path(__file__).resolve().parent.parent / "node_modules" / ".bin"
  413. env["PATH"] = f"{node_bin}{os.pathsep}{env.get('PATH', '')}"
  414. return env
  415. def _child_env(controller: Controller, tool: Tool, overrides: dict[str, str] | None = None) -> dict[str, str]:
  416. return {**_subprocess_env(controller, tool), **(overrides or {})}
  417. def _verify_pinned_webcmd(env: dict[str, str], expected: Path) -> None:
  418. shell = env.get("SHELL") or ("/bin/zsh" if Path("/bin/zsh").is_file() else "/bin/sh")
  419. result = subprocess.run(
  420. [shell, "-lc", "command -v webcmd"],
  421. env=env,
  422. capture_output=True,
  423. text=True,
  424. timeout=10,
  425. )
  426. actual = result.stdout.strip()
  427. if result.returncode or not actual or Path(actual).resolve() != expected.resolve():
  428. raise RuntimeError(
  429. f"Webcmd login shell resolved {actual or 'nothing'}, expected {expected}"
  430. )
  431. def _pin_webcmd_env(base_env: dict[str, str], work_dir: Path) -> dict[str, str]:
  432. executable = shutil.which("webcmd", path=base_env.get("PATH"))
  433. if not executable:
  434. raise RuntimeError("webcmd was not found in PATH")
  435. bin_dir = work_dir / "webcmd-bin"
  436. bin_dir.mkdir()
  437. shim = bin_dir / "webcmd"
  438. shim.write_text(
  439. f"#!/bin/sh\nexec {shlex.quote(str(Path(executable).resolve()))} \"$@\"\n",
  440. encoding="utf-8",
  441. )
  442. shim.chmod(0o700)
  443. zsh_config = work_dir / "zsh-config"
  444. zsh_config.mkdir()
  445. pinned_path = f"{bin_dir}{os.pathsep}{base_env.get('PATH', '')}"
  446. (zsh_config / ".zprofile").write_text(
  447. f"export PATH={shlex.quote(pinned_path)}\n", encoding="utf-8"
  448. )
  449. env = {
  450. **base_env,
  451. "PATH": pinned_path,
  452. "ZDOTDIR": str(zsh_config),
  453. }
  454. _verify_pinned_webcmd(env, shim)
  455. return env
  456. def _session_name(work_dir: Path) -> str:
  457. readable = re.sub(r"[^a-z0-9-]", "-", work_dir.name.lower()).strip("-") or "attempt"
  458. digest = hashlib.sha256(str(work_dir.resolve()).encode()).hexdigest()[:10]
  459. return f"{readable[:37]}-{digest}"
  460. def _short(value: object, limit: int = 2000) -> str:
  461. if isinstance(value, str):
  462. text = value
  463. else:
  464. text = json.dumps(value, default=str, separators=(",", ":"))
  465. return text.strip()[:limit]
  466. def _evidence_excerpt(value: object, limit: int) -> str:
  467. marker = "[truncated]"
  468. text = value if isinstance(value, str) else json.dumps(value, default=str, separators=(",", ":"))
  469. text = text.strip()
  470. if len(text) <= limit:
  471. return text
  472. return text[: limit - len(marker)] + marker
  473. def _command_step(command: str, output: object) -> str:
  474. return (
  475. f"command: {_evidence_excerpt(command, 2000)}\n"
  476. f"output: {_evidence_excerpt(output, 8000)}"
  477. )
  478. def _mcp_content(result: object) -> list[dict]:
  479. if not isinstance(result, dict):
  480. return []
  481. content = result.get("content")
  482. if isinstance(content, list):
  483. return [item for item in content if isinstance(item, dict)]
  484. ok = result.get("Ok")
  485. if isinstance(ok, dict) and isinstance(ok.get("content"), list):
  486. return [item for item in ok["content"] if isinstance(item, dict)]
  487. return []
  488. def _parse_events(
  489. controller: Controller,
  490. lines: list[str],
  491. *,
  492. tool: Tool | None = None,
  493. model: str | None = None,
  494. ) -> ParsedEvents:
  495. steps: list[str] = []
  496. commands: list[str] = []
  497. event_types: list[str] = []
  498. final_fragments: list[str] = []
  499. steps_count = 0
  500. tool_calls = 0
  501. ordinary_input = 0
  502. cache_read_input = 0
  503. cache_creation_input = 0
  504. output_tokens = 0
  505. reasoning_output = 0
  506. usage_seen = False
  507. estimated_api_cost_usd = 0.0
  508. cost_complete = controller == "pi" or (
  509. controller == "codex" and model in GPT_5_6_SOL_MODELS
  510. )
  511. provider_turns = None
  512. provider_duration_seconds = None
  513. provider_api_duration_seconds = None
  514. agent_turns = 0 if controller == "pi" else None
  515. mcp_calls: list[tuple[str, str]] = []
  516. screenshot_images: list[bytes] = []
  517. for line in lines:
  518. try:
  519. event = json.loads(line)
  520. except json.JSONDecodeError:
  521. steps.append(_short(f"raw: {line}"))
  522. continue
  523. event_type = str(event.get("type", "unknown"))
  524. event_types.append(event_type)
  525. if controller == "codex" and event_type == "item.completed":
  526. item = event.get("item") or {}
  527. item_type = item.get("type")
  528. if item_type == "agent_message":
  529. text = str(item.get("text") or "")
  530. final_fragments.append(text)
  531. steps.append(_short(f"text: {text}"))
  532. if text:
  533. steps_count += 1
  534. elif item_type == "command_execution":
  535. command = item.get("command") or ""
  536. command = " ".join(map(str, command)) if isinstance(command, list) else str(command)
  537. commands.append(command)
  538. tool_calls += 1
  539. steps_count += 1
  540. command_output = item.get("aggregated_output") or item.get("output") or ""
  541. steps.append(_command_step(command, command_output))
  542. elif item_type in {"mcp_tool_call", "web_search"}:
  543. event_types.append(str(item_type))
  544. steps.append(_short(f"{item_type}: {_short(item)}"))
  545. steps_count += 1
  546. if item_type == "mcp_tool_call":
  547. server = str(item.get("server") or "")
  548. tool_name = str(item.get("tool") or "")
  549. mcp_calls.append((server, tool_name))
  550. if tool == "libretto":
  551. tool_calls += 1
  552. if (
  553. server == "libretto"
  554. and tool_name == "browser_snapshot"
  555. ):
  556. for content in _mcp_content(item.get("result")):
  557. if (
  558. content.get("type") == "image"
  559. and content.get("mimeType") == "image/png"
  560. and isinstance(content.get("data"), str)
  561. ):
  562. try:
  563. screenshot_images.append(
  564. base64.b64decode(
  565. content["data"], validate=True
  566. )
  567. )
  568. except ValueError:
  569. pass
  570. elif item_type == "reasoning":
  571. text = item.get("text") or item.get("summary") or ""
  572. steps.append(_short(f"reasoning: {text}"))
  573. steps_count += 1
  574. elif controller == "codex" and event_type == "turn.completed":
  575. usage = event.get("usage") or {}
  576. turn_input = int(usage.get("input_tokens") or 0)
  577. turn_cached = int(usage.get("cached_input_tokens") or 0)
  578. turn_cache_write = int(usage.get("cache_write_input_tokens") or 0)
  579. turn_output = int(usage.get("output_tokens") or 0)
  580. ordinary_input += turn_input - turn_cached - turn_cache_write
  581. cache_read_input += turn_cached
  582. cache_creation_input += turn_cache_write
  583. output_tokens += turn_output
  584. reasoning_output += int(usage.get("reasoning_output_tokens") or 0)
  585. usage_seen = usage_seen or bool(usage)
  586. if cost_complete:
  587. if "cache_write_input_tokens" not in usage:
  588. cost_complete = False
  589. else:
  590. input_multiplier = 2.0 if turn_input > LONG_CONTEXT_INPUT_THRESHOLD else 1.0
  591. output_multiplier = 1.5 if turn_input > LONG_CONTEXT_INPUT_THRESHOLD else 1.0
  592. turn_ordinary = turn_input - turn_cached - turn_cache_write
  593. estimated_api_cost_usd += (
  594. input_multiplier
  595. * (
  596. turn_ordinary * GPT_5_6_SOL_PRICES_PER_MILLION["input"]
  597. + turn_cached * GPT_5_6_SOL_PRICES_PER_MILLION["cached_input"]
  598. + turn_cache_write * GPT_5_6_SOL_PRICES_PER_MILLION["cache_write_input"]
  599. )
  600. + output_multiplier
  601. * turn_output
  602. * GPT_5_6_SOL_PRICES_PER_MILLION["output"]
  603. ) / 1_000_000
  604. elif controller == "pi" and event_type == "message_end":
  605. message = event.get("message") or {}
  606. if message.get("role") != "assistant":
  607. continue
  608. agent_turns += 1
  609. usage = message.get("usage") or {}
  610. if usage:
  611. ordinary_input += int(usage.get("input") or 0)
  612. cache_read_input += int(usage.get("cacheRead") or 0)
  613. cache_creation_input += int(usage.get("cacheWrite") or 0)
  614. output_tokens += int(usage.get("output") or 0)
  615. usage_seen = True
  616. turn_cost = (usage.get("cost") or {}).get("total")
  617. if (
  618. isinstance(turn_cost, (int, float))
  619. and not isinstance(turn_cost, bool)
  620. and math.isfinite(turn_cost)
  621. ):
  622. estimated_api_cost_usd += float(turn_cost)
  623. else:
  624. cost_complete = False
  625. else:
  626. cost_complete = False
  627. for block in message.get("content", []) or []:
  628. block_type = block.get("type")
  629. if block_type == "text":
  630. text = str(block.get("text") or "")
  631. final_fragments.append(text)
  632. steps.append(_short(f"text: {text}"))
  633. if text:
  634. steps_count += 1
  635. elif block_type == "thinking":
  636. text = str(block.get("thinking") or "")
  637. if text:
  638. steps.append(_short(f"thinking: {text}"))
  639. steps_count += 1
  640. elif controller == "pi" and event_type == "tool_execution_start":
  641. name = str(event.get("toolName") or "")
  642. arguments = event.get("args") or {}
  643. if name == "bash":
  644. command = str(arguments.get("command") or "")
  645. if command:
  646. commands.append(command)
  647. tool_calls += 1
  648. steps_count += 1
  649. steps.append(_short(f"tool: {name} {_short(arguments)}"))
  650. elif (
  651. name == "read"
  652. and (
  653. _pi_setup_skill_read_allowed(
  654. tool, arguments.get("path")
  655. )
  656. or _pi_agent_browser_screenshot_read_allowed(
  657. tool, arguments.get("path")
  658. )
  659. )
  660. ):
  661. steps.append(_short(f"setup_tool: {name} {_short(arguments)}"))
  662. elif tool == "libretto":
  663. event_types.append("mcp_tool_call")
  664. mcp_calls.append(("libretto", name))
  665. tool_calls += 1
  666. steps_count += 1
  667. steps.append(_short(f"tool: {name} {_short(arguments)}"))
  668. else:
  669. event_types.append("mcp_tool_call")
  670. steps_count += 1
  671. steps.append(_short(f"tool: {name} {_short(arguments)}"))
  672. elif controller == "pi" and event_type == "tool_execution_end":
  673. result = event.get("result") or {}
  674. content = result.get("content") if isinstance(result, dict) else result
  675. steps.append(_short(f"tool_result: {_short(content or '')}"))
  676. if tool == "libretto" and event.get("toolName") == "browser_snapshot":
  677. encoded = None
  678. for item in content if isinstance(content, list) else []:
  679. if (
  680. isinstance(item, dict)
  681. and item.get("type") == "image"
  682. and item.get("mimeType") == "image/png"
  683. and isinstance(item.get("data"), str)
  684. and not item["data"].startswith("[omitted ")
  685. ):
  686. encoded = item["data"]
  687. break
  688. details = result.get("details") if isinstance(result, dict) else None
  689. screenshot = details.get("screenshot") if isinstance(details, dict) else None
  690. if encoded is None and isinstance(screenshot, dict):
  691. if screenshot.get("mimeType") == "image/png":
  692. encoded = screenshot.get("base64")
  693. if isinstance(encoded, str):
  694. try:
  695. screenshot_images.append(
  696. base64.b64decode(encoded, validate=True)
  697. )
  698. except ValueError:
  699. pass
  700. elif controller == "pi" and event_type == "result":
  701. text = str(event.get("result") or "")
  702. if text:
  703. final_fragments.append(text)
  704. steps.append(_short(f"result: {text}"))
  705. if event.get("duration_ms") is not None:
  706. provider_duration_seconds = float(event["duration_ms"]) / 1000
  707. elif controller == "claude" and event_type in {"assistant", "user"}:
  708. message = event.get("message") or {}
  709. usage = message.get("usage") or {}
  710. if usage:
  711. ordinary_input += int(usage.get("input_tokens") or 0)
  712. cache_read_input += int(usage.get("cache_read_input_tokens") or 0)
  713. cache_creation_input += int(usage.get("cache_creation_input_tokens") or 0)
  714. output_tokens += int(usage.get("output_tokens") or 0)
  715. usage_seen = True
  716. for block in message.get("content", []) or []:
  717. block_type = block.get("type")
  718. if block_type == "tool_use":
  719. name = block.get("name")
  720. command = str((block.get("input") or {}).get("command") or "")
  721. if name == "Bash" and command:
  722. commands.append(command)
  723. tool_calls += 1
  724. else:
  725. event_types.append("mcp_tool_call")
  726. steps.append(_short(f"tool: {_short(block)}"))
  727. steps_count += 1
  728. elif block_type == "tool_result":
  729. steps.append(_short(f"tool_result: {_short(block.get('content') or '')}"))
  730. steps_count += 1
  731. elif block_type == "text":
  732. text = str(block.get("text") or "")
  733. final_fragments.append(text)
  734. steps.append(_short(f"text: {text}"))
  735. if text:
  736. steps_count += 1
  737. elif block_type == "thinking":
  738. text = str(block.get("thinking") or "")
  739. if text:
  740. steps.append(_short(f"thinking: {text}"))
  741. steps_count += 1
  742. elif controller == "claude" and event_type == "result":
  743. text = str(event.get("result") or "")
  744. if text:
  745. final_fragments.append(text)
  746. steps.append(_short(f"result: {text}"))
  747. usage = event.get("usage") or {}
  748. if usage and not usage_seen:
  749. ordinary_input += int(usage.get("input_tokens") or 0)
  750. cache_read_input += int(usage.get("cache_read_input_tokens") or 0)
  751. cache_creation_input += int(usage.get("cache_creation_input_tokens") or 0)
  752. output_tokens += int(usage.get("output_tokens") or 0)
  753. usage_seen = True
  754. if event.get("num_turns") is not None:
  755. provider_turns = int(event["num_turns"])
  756. if event.get("duration_ms") is not None:
  757. provider_duration_seconds = float(event["duration_ms"]) / 1000
  758. if event.get("duration_api_ms") is not None:
  759. provider_api_duration_seconds = float(event["duration_api_ms"]) / 1000
  760. elif event_type not in {"thread.started", "turn.started", "turn.completed", "system"}:
  761. steps.append(_short(f"{event_type}: {_short(event)}"))
  762. tokens = None
  763. if usage_seen:
  764. input_tokens = ordinary_input + cache_read_input + cache_creation_input
  765. tokens = TokenUsage(
  766. input=input_tokens,
  767. cache_read_input=cache_read_input,
  768. cache_creation_input=cache_creation_input,
  769. non_cached_input=ordinary_input,
  770. output=output_tokens,
  771. reasoning_output=reasoning_output if controller == "codex" else None,
  772. total=input_tokens + output_tokens,
  773. estimated_api_cost_usd=(
  774. estimated_api_cost_usd if cost_complete else None
  775. ),
  776. )
  777. return ParsedEvents(
  778. steps=steps,
  779. commands=commands,
  780. event_types=event_types,
  781. final_text="\n".join(final_fragments),
  782. steps_count=steps_count,
  783. tool_calls=tool_calls,
  784. tokens=tokens,
  785. provider_turns=provider_turns,
  786. provider_duration_seconds=provider_duration_seconds,
  787. provider_api_duration_seconds=provider_api_duration_seconds,
  788. agent_turns=agent_turns,
  789. mcp_calls=mcp_calls,
  790. screenshot_images=screenshot_images,
  791. )
  792. def _extract_final_answer(text: str) -> str:
  793. matches = FINAL_ANSWER_RE.findall(text)
  794. return matches[-1].strip() if matches else ""
  795. def _has_unescaped_dollar(command: str) -> bool:
  796. def starts_expansion(position: int) -> bool:
  797. if position + 1 >= len(command):
  798. return False
  799. following = command[position + 1]
  800. return following.isalnum() or following == "_" or following in "{(?$#*@!-"
  801. quote: str | None = None
  802. index = 0
  803. while index < len(command):
  804. character = command[index]
  805. if quote == "'":
  806. if character == "'":
  807. quote = None
  808. index += 1
  809. elif quote == '"':
  810. if character == '"':
  811. quote = None
  812. index += 1
  813. elif character == "\\" and index + 1 < len(command) and command[index + 1] in '$`"\\\n':
  814. index += 2
  815. elif character == "$" and starts_expansion(index):
  816. return True
  817. else:
  818. index += 1
  819. elif character in "'\"":
  820. quote = character
  821. index += 1
  822. elif character == "\\":
  823. index += 2
  824. elif character == "$" and starts_expansion(index):
  825. return True
  826. else:
  827. index += 1
  828. return False
  829. def _shell_segments(command: str) -> list[list[str]] | None:
  830. try:
  831. lexer = shlex.shlex(command, posix=True, punctuation_chars=";&|()<>\n`")
  832. lexer.whitespace = " \t\r"
  833. lexer.whitespace_split = True
  834. lexer.commenters = ""
  835. tokens = list(lexer)
  836. except ValueError:
  837. return None
  838. controls = {";", "&&", "||", "|", "&", "\n"}
  839. punctuation = set(";&|()<>\n`")
  840. segments: list[list[str]] = []
  841. segment: list[str] = []
  842. for token in tokens:
  843. if token in controls:
  844. if not segment:
  845. return None
  846. segments.append(segment)
  847. segment = []
  848. elif token and set(token) <= punctuation:
  849. return None
  850. else:
  851. segment.append(token)
  852. if not segment and segments:
  853. return None
  854. if segment:
  855. segments.append(segment)
  856. return segments
  857. def _strip_shell_quotes(value: str) -> str:
  858. if len(value) >= 2 and value[0] == value[-1] and value[0] in {"'", '"'}:
  859. return value[1:-1]
  860. return value
  861. def _is_unterminated_allowed_command(tool: Tool, command: str) -> bool:
  862. try:
  863. shlex.split(command, posix=True)
  864. except ValueError:
  865. words = command.lstrip().split()
  866. if not words:
  867. return False
  868. executable = Path(words[0].strip("'\"")).name
  869. if tool in {"webcmd", "agent-browser", "browser-use", "playwright-cli"}:
  870. return executable == tool
  871. if tool == "chrome-devtools-axi":
  872. return (
  873. executable == "npx"
  874. and len(words) >= 3
  875. and words[1:3] == ["-y", "chrome-devtools-axi"]
  876. )
  877. return False
  878. def _dev_browser_segment_allowed(segment: list[str]) -> bool:
  879. if not segment or segment[0] != "dev-browser":
  880. return False
  881. arguments = segment[1:]
  882. if not arguments:
  883. return True
  884. if len(arguments) == 2 and arguments[0] == "--timeout":
  885. return arguments[1].isdigit() and int(arguments[1]) > 0
  886. if len(arguments) == 1 and arguments[0].startswith("--timeout="):
  887. value = arguments[0].partition("=")[2]
  888. return value.isdigit() and int(value) > 0
  889. return False
  890. def _browser_use_segment_allowed(segment: list[str]) -> bool:
  891. return bool(segment) and segment == ["browser-use"]
  892. def _dev_browser_skill_read_allowed(command: str) -> bool:
  893. segments = _shell_segments(command)
  894. if segments is None or len(segments) != 1:
  895. return False
  896. segment = segments[0]
  897. skill = (Path.home() / ".codex/skills/dev-browser/SKILL.md").resolve()
  898. try:
  899. target = Path(segment[-1]).expanduser().resolve()
  900. except (IndexError, OSError):
  901. return False
  902. return target == skill and segment[:-1] in (
  903. ["cat"],
  904. ["sed", "-n", "1,240p"],
  905. )
  906. def _browser_use_skill_read_allowed(command: str) -> bool:
  907. segments = _shell_segments(command)
  908. if segments is None or len(segments) != 1:
  909. return False
  910. segment = segments[0]
  911. skill = (BROWSER_USE_SKILL / "SKILL.md").resolve()
  912. try:
  913. target = Path(segment[-1]).expanduser().resolve()
  914. except (IndexError, OSError):
  915. return False
  916. return target == skill and segment[:-1] in (
  917. ["cat"],
  918. ["sed", "-n", "1,240p"],
  919. )
  920. def _unwrap_single_shell_command(command: str) -> str | None:
  921. stripped = command.lstrip()
  922. first = stripped.split(None, 1)[0] if stripped else ""
  923. if Path(first.strip("'\"")).name not in {"sh", "bash", "zsh"}:
  924. return command
  925. if _has_unescaped_dollar(command):
  926. return None
  927. try:
  928. words = shlex.split(command, posix=True)
  929. except ValueError:
  930. return None
  931. if (
  932. len(words) != 3
  933. or Path(words[0]).name not in {"sh", "bash", "zsh"}
  934. or words[1] not in {"-c", "-lc"}
  935. ):
  936. return None
  937. return words[2]
  938. def _webcmd_skill_path_allowed(value: object) -> bool:
  939. try:
  940. target = Path(str(value or "")).expanduser().resolve()
  941. target.relative_to(WEBCMD_BROWSER_SKILL_ROOT)
  942. except (OSError, ValueError):
  943. return False
  944. return target.is_file()
  945. def _pi_setup_skill_read_allowed(tool: Tool | None, value: object) -> bool:
  946. selected_tool = tool or "webcmd"
  947. if selected_tool == "webcmd":
  948. return _webcmd_skill_path_allowed(value)
  949. try:
  950. target = Path(str(value or "")).expanduser().resolve()
  951. except OSError:
  952. return False
  953. return target in PI_SETUP_SKILL_FILES.get(selected_tool, frozenset())
  954. def _pi_agent_browser_screenshot_read_allowed(
  955. tool: Tool | None, value: object
  956. ) -> bool:
  957. if tool not in {"agent-browser", "playwright-cli"}:
  958. return False
  959. try:
  960. target = Path(str(value or "")).expanduser().resolve()
  961. except OSError:
  962. return False
  963. return (
  964. target.suffix.lower() == ".png"
  965. and target.is_file()
  966. and "shots" in target.parts
  967. )
  968. def _webcmd_skill_read_allowed(command: str) -> bool:
  969. inner = _unwrap_single_shell_command(command)
  970. if inner is None:
  971. return False
  972. segments = _shell_segments(inner)
  973. if segments is None or len(segments) != 1:
  974. return False
  975. segment = segments[0]
  976. if len(segment) == 2 and segment[0] == "cat":
  977. target_value = segment[1]
  978. elif (
  979. len(segment) == 4
  980. and segment[:2] == ["sed", "-n"]
  981. and re.fullmatch(r"[1-9][0-9]*,[1-9][0-9]*p", segment[2]) is not None
  982. ):
  983. target_value = segment[3]
  984. else:
  985. return False
  986. return _webcmd_skill_path_allowed(target_value)
  987. def _dev_browser_command_allowed(command: str) -> bool:
  988. inner = _unwrap_single_shell_command(command)
  989. if inner is None:
  990. return False
  991. if _dev_browser_skill_read_allowed(inner):
  992. return True
  993. lines = inner.splitlines()
  994. if len(lines) < 3:
  995. segments = _shell_segments(inner)
  996. return (
  997. segments is not None
  998. and len(segments) == 1
  999. and _dev_browser_segment_allowed(segments[0])
  1000. )
  1001. match = re.fullmatch(
  1002. r"[ \t]*(?P<header>dev-browser(?:[ \t]+.*?)?)[ \t]+"
  1003. r"<<[ \t]*(?P<quote>['\"])(?P<label>[A-Za-z_][A-Za-z0-9_]*)"
  1004. r"(?P=quote)[ \t]*",
  1005. lines[0],
  1006. )
  1007. if match is None or lines[-1].rstrip("\r") != match.group("label"):
  1008. return False
  1009. header = match.group("header")
  1010. if _has_unescaped_dollar(header):
  1011. return False
  1012. try:
  1013. segment = shlex.split(header, posix=True)
  1014. except ValueError:
  1015. return False
  1016. return _dev_browser_segment_allowed(segment)
  1017. def _browser_use_command_allowed(command: str) -> bool:
  1018. inner = _unwrap_single_shell_command(command)
  1019. if inner is None:
  1020. return False
  1021. if _browser_use_skill_read_allowed(inner):
  1022. return True
  1023. lines = inner.splitlines()
  1024. if len(lines) < 3:
  1025. segments = _shell_segments(inner)
  1026. return (
  1027. segments is not None
  1028. and len(segments) == 1
  1029. and _browser_use_segment_allowed(segments[0])
  1030. )
  1031. match = re.fullmatch(
  1032. r"[ \t]*(?P<header>browser-use(?:[ \t]+.*?)?)[ \t]+"
  1033. r"<<[ \t]*(?P<quote>['\"])(?P<label>[A-Za-z_][A-Za-z0-9_]*)"
  1034. r"(?P=quote)[ \t]*",
  1035. lines[0],
  1036. )
  1037. if match is None or lines[-1].rstrip("\r") != match.group("label"):
  1038. return False
  1039. header = match.group("header")
  1040. if _has_unescaped_dollar(header):
  1041. return False
  1042. try:
  1043. segment = shlex.split(header, posix=True)
  1044. except ValueError:
  1045. return False
  1046. return _browser_use_segment_allowed(segment)
  1047. def _webcmd_run_heredoc_allowed(command: str) -> bool:
  1048. inner = _unwrap_single_shell_command(command)
  1049. if inner is None:
  1050. return False
  1051. lines = inner.splitlines()
  1052. if len(lines) < 3:
  1053. return False
  1054. match = re.fullmatch(
  1055. r"[ \t]*(?P<header>webcmd[ \t]+[^<\r\n]+)[ \t]+"
  1056. r"<<[ \t]*"
  1057. r"(?P<quote>['\"])(?P<label>[A-Za-z_][A-Za-z0-9_]*)"
  1058. r"(?P=quote)[ \t]*",
  1059. lines[0],
  1060. )
  1061. if match is None or lines[-1].rstrip("\r") != match.group("label"):
  1062. return False
  1063. if any(
  1064. line.rstrip("\r") == match.group("label")
  1065. for line in lines[1:-1]
  1066. ):
  1067. return False
  1068. header = match.group("header")
  1069. if _has_unescaped_dollar(header):
  1070. return False
  1071. try:
  1072. segment = shlex.split(header, posix=True)
  1073. except ValueError:
  1074. return False
  1075. browser = _webcmd_browser_tail(segment)
  1076. if browser is None or len(browser) < 2 or browser[:2] != ["browser", "run"]:
  1077. return False
  1078. seen: set[str] = set()
  1079. arguments = browser[2:]
  1080. index = 0
  1081. while index < len(arguments):
  1082. option = arguments[index]
  1083. if option in seen:
  1084. return False
  1085. seen.add(option)
  1086. if option == "--stdin":
  1087. index += 1
  1088. continue
  1089. if option == "--no-snapshot-diff":
  1090. index += 1
  1091. continue
  1092. if option not in {"--timeout", "--max-output", "--snapshot-mode"}:
  1093. return False
  1094. if index + 1 >= len(arguments):
  1095. return False
  1096. value = arguments[index + 1]
  1097. if option in {"--timeout", "--max-output"}:
  1098. if not value.isdigit() or int(value) <= 0:
  1099. return False
  1100. elif value not in {"act", "tree"}:
  1101. return False
  1102. index += 2
  1103. return "--stdin" in seen
  1104. def _webcmd_browser_tail(segment: list[str]) -> list[str] | None:
  1105. if (
  1106. len(segment) < 7
  1107. or segment[:3] != ["webcmd", "--profile", WEBCMD_BENCHMARK_PROFILE]
  1108. or segment[3] != "--session"
  1109. or WEBCMD_SESSION_RE.fullmatch(segment[4]) is None
  1110. or segment[5] != "browser"
  1111. ):
  1112. return None
  1113. return segment[5:]
  1114. def _webcmd_snapshot_segment_allowed(segment: list[str]) -> bool:
  1115. browser = _webcmd_browser_tail(segment)
  1116. if browser is None or len(browser) < 2 or browser[:2] != ["browser", "snapshot"]:
  1117. return False
  1118. arguments = browser[2:]
  1119. seen: set[str] = set()
  1120. index = 0
  1121. while index < len(arguments):
  1122. option = arguments[index]
  1123. if option in seen:
  1124. return False
  1125. seen.add(option)
  1126. if option not in {"--snapshot-mode", "--max-output", "--ref"}:
  1127. return False
  1128. if index + 1 >= len(arguments):
  1129. return False
  1130. value = arguments[index + 1]
  1131. if option == "--snapshot-mode" and value not in {"act", "tree", "read"}:
  1132. return False
  1133. if option == "--max-output" and (not value.isdigit() or int(value) <= 0):
  1134. return False
  1135. if option == "--ref" and re.fullmatch(r"[A-Za-z0-9._:-]+", value) is None:
  1136. return False
  1137. index += 2
  1138. return True
  1139. def _webcmd_segment_allowed(segment: list[str]) -> bool:
  1140. browser = _webcmd_browser_tail(segment)
  1141. if browser is None:
  1142. return False
  1143. if browser == ["browser", "tabs"]:
  1144. return True
  1145. if _webcmd_snapshot_segment_allowed(segment):
  1146. return True
  1147. return (
  1148. len(browser) == 4
  1149. and browser[1:3] == ["bind", "--page"]
  1150. and re.fullmatch(r"[A-Za-z0-9._:-]+", browser[3]) is not None
  1151. )
  1152. def _playwright_cli_segment_allowed(segment: list[str]) -> bool:
  1153. executable = Path(segment[0].strip("'\"")).name
  1154. if executable != "playwright-cli" or len(segment) < 2:
  1155. return False
  1156. arguments = segment[1:]
  1157. forbidden_options = {
  1158. "--browser", "--cdp", "--config", "--endpoint", "--extension",
  1159. "--persistent", "--profile", "--session", "-s",
  1160. }
  1161. if any(
  1162. argument in forbidden_options
  1163. or argument.startswith("-s=")
  1164. or any(argument.startswith(f"{option}=") for option in forbidden_options)
  1165. for argument in arguments
  1166. ):
  1167. return False
  1168. positional = [argument for argument in arguments if not argument.startswith("-")]
  1169. if not positional:
  1170. return False
  1171. return positional[0] not in {
  1172. "attach", "close", "close-all", "delete-data", "detach", "install",
  1173. "install-browser", "kill-all", "mcp", "show",
  1174. }
  1175. def _segment_allowed(tool: Tool, segment: list[str]) -> bool:
  1176. executable = Path(segment[0].strip("'\"")).name
  1177. if executable in {"sh", "bash", "zsh"} and len(segment) == 3 and segment[1] in {"-c", "-lc"}:
  1178. inner = _strip_shell_quotes(segment[2])
  1179. return not _policy_violation(tool, [inner], [])
  1180. if tool == "agent-browser":
  1181. if executable != "agent-browser" or len(segment) < 2:
  1182. return False
  1183. arguments = segment[1:]
  1184. forbidden_options = {
  1185. "--args", "--auto-connect", "--cdp", "--config", "--engine",
  1186. "--executable-path", "--namespace", "--profile", "--provider",
  1187. "--restore", "--session", "--state",
  1188. }
  1189. if any(
  1190. argument in forbidden_options
  1191. or any(argument.startswith(f"{option}=") for option in forbidden_options)
  1192. for argument in arguments
  1193. ):
  1194. return False
  1195. if arguments[0] == "read" and any(
  1196. argument.startswith(("http://", "https://"))
  1197. for argument in arguments[1:]
  1198. ):
  1199. return False
  1200. return arguments[0] not in {
  1201. "batch", "close", "connect", "doctor", "install", "mcp", "plugin",
  1202. "upgrade",
  1203. }
  1204. if tool == "playwright-cli":
  1205. return _playwright_cli_segment_allowed(segment)
  1206. if tool == "dev-browser":
  1207. return _dev_browser_segment_allowed(segment)
  1208. if tool == "browser-use":
  1209. return _browser_use_segment_allowed(segment)
  1210. if tool == "webcmd":
  1211. return _webcmd_segment_allowed(segment)
  1212. if executable == tool:
  1213. return False
  1214. if tool == "chrome-devtools-axi" and executable == "npx":
  1215. return len(segment) >= 3 and segment[1:3] == ["-y", "chrome-devtools-axi"]
  1216. return False
  1217. def _policy_violation(
  1218. tool: Tool,
  1219. commands: list[str],
  1220. event_types: list[str],
  1221. mcp_calls: list[tuple[str, str]] | None = None,
  1222. ) -> bool:
  1223. if tool == "libretto":
  1224. calls = mcp_calls or []
  1225. return (
  1226. bool(commands)
  1227. or "web_search" in event_types
  1228. or event_types.count("mcp_tool_call") != len(calls)
  1229. or any(
  1230. server != "libretto" or name not in LIBRETTO_TOOLS
  1231. for server, name in calls
  1232. )
  1233. )
  1234. if any(event in {"web_search", "mcp_tool_call"} for event in event_types):
  1235. return True
  1236. for command in commands:
  1237. if tool == "dev-browser":
  1238. if not _dev_browser_command_allowed(command):
  1239. return True
  1240. continue
  1241. if tool == "browser-use":
  1242. if not _browser_use_command_allowed(command):
  1243. return True
  1244. continue
  1245. if tool == "webcmd" and _webcmd_run_heredoc_allowed(command):
  1246. continue
  1247. if tool == "webcmd" and _webcmd_skill_read_allowed(command):
  1248. continue
  1249. if _has_unescaped_dollar(command):
  1250. return True
  1251. segments = _shell_segments(command)
  1252. if segments is None and _is_unterminated_allowed_command(tool, command):
  1253. continue
  1254. if (
  1255. segments is None
  1256. or (tool in {"agent-browser", "playwright-cli"} and len(segments) != 1)
  1257. or any(not _segment_allowed(tool, segment) for segment in segments)
  1258. ):
  1259. return True
  1260. return False
  1261. def _kill_process_group(process: asyncio.subprocess.Process) -> None:
  1262. try:
  1263. os.killpg(process.pid, signal.SIGKILL)
  1264. except (AttributeError, ProcessLookupError, PermissionError):
  1265. try:
  1266. process.kill()
  1267. except ProcessLookupError:
  1268. pass
  1269. async def _bounded_drain_and_reap(process: asyncio.subprocess.Process) -> tuple[bytes, bytes]:
  1270. try:
  1271. return await asyncio.wait_for(process.communicate(), timeout=CLEANUP_TIMEOUT_SECONDS)
  1272. except BaseException:
  1273. try:
  1274. await asyncio.wait_for(process.wait(), timeout=CLEANUP_TIMEOUT_SECONDS)
  1275. except BaseException:
  1276. pass
  1277. return b"", b""
  1278. async def _terminate_controller(process: asyncio.subprocess.Process) -> tuple[bytes, bytes]:
  1279. _kill_process_group(process)
  1280. cleanup = asyncio.create_task(_bounded_drain_and_reap(process))
  1281. while True:
  1282. try:
  1283. return await asyncio.shield(cleanup)
  1284. except asyncio.CancelledError:
  1285. if cleanup.done():
  1286. return cleanup.result()
  1287. async def _close_session_cancellation_resistant(tool: Tool, session: str, tool_env: dict[str, str] | None = None) -> asyncio.CancelledError | None:
  1288. close = asyncio.create_task(_close_session(tool, session, tool_env))
  1289. cancellation: asyncio.CancelledError | None = None
  1290. while True:
  1291. try:
  1292. await asyncio.shield(close)
  1293. return cancellation
  1294. except asyncio.CancelledError as error:
  1295. if cancellation is None:
  1296. cancellation = error
  1297. if close.cancelled():
  1298. return cancellation
  1299. except Exception as cleanup_error:
  1300. if cancellation is None:
  1301. raise
  1302. cancellation.add_note(f"Webcmd cleanup also failed: {cleanup_error}")
  1303. return cancellation
  1304. async def _create_webcmd_session(tool_env: dict[str, str] | None = None) -> str:
  1305. command = [
  1306. "webcmd",
  1307. "--profile",
  1308. WEBCMD_BENCHMARK_PROFILE,
  1309. "session",
  1310. "create",
  1311. "-f",
  1312. "json",
  1313. ]
  1314. env = {**_subprocess_env(tool="webcmd"), **(tool_env or {})}
  1315. process = await asyncio.create_subprocess_exec(
  1316. *command,
  1317. env=env,
  1318. stdout=asyncio.subprocess.PIPE,
  1319. stderr=asyncio.subprocess.PIPE,
  1320. start_new_session=True,
  1321. )
  1322. try:
  1323. stdout, stderr = await asyncio.wait_for(
  1324. process.communicate(), timeout=60
  1325. )
  1326. except asyncio.TimeoutError:
  1327. _kill_process_group(process)
  1328. await process.wait()
  1329. raise RuntimeError("Webcmd Session creation timed out")
  1330. if process.returncode:
  1331. message = stderr.decode(errors="replace").strip()
  1332. raise RuntimeError(f"Webcmd Session creation failed: {message}")
  1333. try:
  1334. payload = json.loads(stdout)
  1335. session = payload["id"]
  1336. except (json.JSONDecodeError, KeyError, TypeError) as error:
  1337. raise RuntimeError("Webcmd Session creation returned invalid JSON") from error
  1338. if not isinstance(session, str) or WEBCMD_SESSION_RE.fullmatch(session) is None:
  1339. raise RuntimeError("Webcmd Session creation returned an invalid Session ID")
  1340. return session
  1341. async def _close_runtime_cancellation_resistant(runtime) -> asyncio.CancelledError | None:
  1342. close = asyncio.create_task(runtime.close())
  1343. cancellation: asyncio.CancelledError | None = None
  1344. while True:
  1345. try:
  1346. await asyncio.shield(close)
  1347. return cancellation
  1348. except asyncio.CancelledError as error:
  1349. if cancellation is None:
  1350. cancellation = error
  1351. if close.cancelled():
  1352. return cancellation
  1353. async def _close_session(tool: Tool, session: str, tool_env: dict[str, str] | None = None) -> None:
  1354. if tool != "webcmd":
  1355. return
  1356. command = [
  1357. "webcmd",
  1358. "--profile",
  1359. WEBCMD_BENCHMARK_PROFILE,
  1360. "session",
  1361. "close",
  1362. session,
  1363. "--force",
  1364. "-f",
  1365. "json",
  1366. ]
  1367. try:
  1368. process = await asyncio.create_subprocess_exec(*command, env={**_subprocess_env(tool=tool), **(tool_env or {})}, stdout=asyncio.subprocess.DEVNULL, stderr=asyncio.subprocess.DEVNULL, start_new_session=True)
  1369. except Exception as error:
  1370. raise RuntimeError("Webcmd Session cleanup failed to start") from error
  1371. try:
  1372. await asyncio.wait_for(process.wait(), timeout=15)
  1373. except asyncio.TimeoutError:
  1374. _kill_process_group(process)
  1375. try:
  1376. await asyncio.wait_for(process.wait(), timeout=CLEANUP_TIMEOUT_SECONDS)
  1377. except asyncio.TimeoutError:
  1378. raise RuntimeError("Webcmd Session cleanup timed out and could not be reaped")
  1379. raise RuntimeError("Webcmd Session cleanup timed out")
  1380. if process.returncode:
  1381. raise RuntimeError(f"Webcmd Session cleanup failed with exit code {process.returncode}")
  1382. WEBCMD_ARTIFACT_LOCATOR_RE = re.compile(
  1383. r"browser-run://(artifact_[0-9a-f]{24})/([^\"'\\\s]+)"
  1384. )
  1385. def _webcmd_result_payloads(controller: Controller, lines: list[str]) -> list[object]:
  1386. payloads: list[object] = []
  1387. for line in lines:
  1388. try:
  1389. event = json.loads(line)
  1390. except json.JSONDecodeError:
  1391. continue
  1392. event_type = event.get("type")
  1393. if controller == "codex" and event_type == "item.completed":
  1394. item = event.get("item") or {}
  1395. command = str(item.get("command") or "")
  1396. if (
  1397. item.get("type") == "command_execution"
  1398. and _webcmd_run_heredoc_allowed(command)
  1399. ):
  1400. payloads.append(item.get("aggregated_output") or item.get("output") or "")
  1401. elif controller == "pi" and event_type == "tool_execution_end":
  1402. payloads.append(event.get("result") or "")
  1403. elif controller == "claude" and event_type in {"assistant", "user"}:
  1404. for block in (event.get("message") or {}).get("content", []) or []:
  1405. if block.get("type") == "tool_result":
  1406. payloads.append(block.get("content") or "")
  1407. return payloads
  1408. def _collect_webcmd_screenshots(
  1409. controller: Controller,
  1410. lines: list[str],
  1411. shots_dir: Path,
  1412. ) -> None:
  1413. locators: list[tuple[str, str]] = []
  1414. seen: set[tuple[str, str]] = set()
  1415. for payload in _webcmd_result_payloads(controller, lines):
  1416. text = payload if isinstance(payload, str) else json.dumps(payload, default=str)
  1417. for match in WEBCMD_ARTIFACT_LOCATOR_RE.finditer(text):
  1418. locator = (match.group(1), unquote(match.group(2)))
  1419. if locator not in seen:
  1420. seen.add(locator)
  1421. locators.append(locator)
  1422. cache_dir = Path.home() / ".webcmd" / "cache" / "browser-run"
  1423. next_index = 1
  1424. for artifact_id, filename in locators:
  1425. logical_path = Path(filename)
  1426. if (
  1427. not filename
  1428. or logical_path.is_absolute()
  1429. or "\\" in filename
  1430. or ".." in logical_path.parts
  1431. ):
  1432. raise ValueError(f"Unsafe Webcmd artifact path: {filename}")
  1433. if logical_path.suffix.lower() != ".png":
  1434. continue
  1435. artifact_dir = (cache_dir / artifact_id).resolve()
  1436. source = (artifact_dir / logical_path).resolve()
  1437. if not source.is_relative_to(artifact_dir):
  1438. raise ValueError(f"Unsafe Webcmd artifact path: {filename}")
  1439. if not source.is_file():
  1440. raise FileNotFoundError(f"Missing Webcmd screenshot artifact: {source}")
  1441. destination = shots_dir / f"step_{next_index:03}.png"
  1442. while destination.exists():
  1443. next_index += 1
  1444. destination = shots_dir / f"step_{next_index:03}.png"
  1445. shutil.copy2(source, destination)
  1446. next_index += 1
  1447. async def run_controller(controller: Controller, model: str, tool: Tool, task: str, work_dir: Path, timeout_seconds: int, reasoning_effort: str | None = None, tool_env: dict[str, str] | None = None) -> ExecutionEvidence:
  1448. work_dir.mkdir(parents=True, exist_ok=False)
  1449. shots_dir = work_dir / "shots"
  1450. shots_dir.mkdir()
  1451. session = _session_name(work_dir)
  1452. browser_runtime = None
  1453. if tool == "chrome-devtools-axi":
  1454. browser_runtime = await start_axi_runtime(session, work_dir, _subprocess_env(tool=tool))
  1455. elif tool == "agent-browser":
  1456. browser_runtime = await start_agent_browser_runtime(session, work_dir, _subprocess_env(tool=tool))
  1457. elif tool == "playwright-cli":
  1458. browser_runtime = await start_playwright_cli_runtime(session, work_dir, _subprocess_env(tool=tool))
  1459. elif tool == "dev-browser":
  1460. browser_runtime = await start_dev_browser_runtime(session, work_dir, _subprocess_env(tool=tool))
  1461. elif tool == "libretto":
  1462. browser_runtime = await start_libretto_runtime(session, work_dir, _subprocess_env(tool=tool))
  1463. elif tool == "browser-use":
  1464. browser_runtime = await start_browser_use_runtime(session, work_dir, _subprocess_env(tool=tool))
  1465. merged_tool_env = {**(browser_runtime.env if browser_runtime else {}), **(tool_env or {})}
  1466. if tool == "webcmd":
  1467. merged_tool_env = await asyncio.to_thread(
  1468. _pin_webcmd_env,
  1469. {**_subprocess_env(tool=tool), **merged_tool_env},
  1470. work_dir,
  1471. )
  1472. session = await _create_webcmd_session(merged_tool_env)
  1473. prompt = _build_prompt(tool, session, shots_dir, task)
  1474. turn_collector = _CodexTurnCollector() if controller == "codex" else None
  1475. if turn_collector is not None:
  1476. turn_collector.__enter__()
  1477. try:
  1478. command, stdin = _controller_command(
  1479. controller,
  1480. model,
  1481. prompt,
  1482. reasoning_effort,
  1483. tool,
  1484. browser_runtime.env if browser_runtime else None,
  1485. otel_endpoint=(
  1486. turn_collector.endpoint if turn_collector is not None else None
  1487. ),
  1488. )
  1489. except BaseException as setup_error:
  1490. if turn_collector is not None:
  1491. turn_collector.__exit__()
  1492. if tool == "webcmd":
  1493. try:
  1494. close_cancellation = await _close_session_cancellation_resistant(
  1495. tool, session, merged_tool_env
  1496. )
  1497. except Exception as cleanup_error:
  1498. setup_error.add_note(f"Webcmd cleanup also failed: {cleanup_error}")
  1499. close_cancellation = None
  1500. if close_cancellation is not None:
  1501. raise close_cancellation
  1502. raise
  1503. timed_out = False
  1504. started = time.monotonic()
  1505. process = None
  1506. pending_error: BaseException | None = None
  1507. pending_traceback = None
  1508. cleanup_error: Exception | None = None
  1509. close_cancellation: asyncio.CancelledError | None = None
  1510. try:
  1511. process = await asyncio.create_subprocess_exec(*command, cwd=work_dir, env=_child_env(controller, tool, merged_tool_env), stdin=asyncio.subprocess.PIPE if stdin is not None else None, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, start_new_session=True)
  1512. try:
  1513. stdout, stderr = await asyncio.wait_for(process.communicate(stdin), timeout=timeout_seconds)
  1514. except asyncio.TimeoutError:
  1515. timed_out = True
  1516. stdout, stderr = await _terminate_controller(process)
  1517. except BaseException as error:
  1518. pending_error = error
  1519. pending_traceback = error.__traceback__
  1520. if process is not None:
  1521. await _terminate_controller(process)
  1522. finally:
  1523. controller_duration = time.monotonic() - started
  1524. try:
  1525. if browser_runtime is None:
  1526. close_cancellation = await _close_session_cancellation_resistant(tool, session, merged_tool_env)
  1527. else:
  1528. close_cancellation = await _close_runtime_cancellation_resistant(browser_runtime)
  1529. except Exception as error:
  1530. cleanup_error = error
  1531. if turn_collector is not None:
  1532. turn_collector.__exit__()
  1533. if pending_error is not None:
  1534. if cleanup_error is not None:
  1535. pending_error.add_note(f"Browser cleanup also failed: {cleanup_error}")
  1536. raise pending_error.with_traceback(pending_traceback)
  1537. if cleanup_error is not None:
  1538. raise cleanup_error
  1539. if close_cancellation is not None:
  1540. raise close_cancellation
  1541. lines = stdout.decode(errors="replace").splitlines()
  1542. (work_dir / "controller.jsonl").write_text("\n".join(lines) + ("\n" if lines else ""), encoding="utf-8")
  1543. if stderr:
  1544. lines.append(json.dumps({"type": "stderr", "text": stderr.decode(errors="replace")[:2000]}))
  1545. parsed = _parse_events(controller, lines, tool=tool, model=model)
  1546. if tool == "webcmd":
  1547. _collect_webcmd_screenshots(controller, lines, shots_dir)
  1548. for index, image in enumerate(parsed.screenshot_images, start=1):
  1549. (shots_dir / f"step_{index:03}.png").write_bytes(image)
  1550. final_answer = _extract_final_answer(parsed.final_text)
  1551. if timed_out:
  1552. termination: Termination = "timeout"
  1553. elif process.returncode != 0:
  1554. termination = "controller_error"
  1555. elif _policy_violation(
  1556. tool, parsed.commands, parsed.event_types, parsed.mcp_calls
  1557. ):
  1558. termination = "tool_policy_violation"
  1559. elif not final_answer:
  1560. termination = "missing_final_answer"
  1561. else:
  1562. termination = "completed"
  1563. metrics = ControllerMetrics(
  1564. duration_seconds=controller_duration,
  1565. steps=parsed.steps_count,
  1566. tool_calls=parsed.tool_calls,
  1567. tokens=parsed.tokens,
  1568. provider_turns=parsed.provider_turns,
  1569. provider_duration_seconds=parsed.provider_duration_seconds,
  1570. provider_api_duration_seconds=parsed.provider_api_duration_seconds,
  1571. agent_turns=(
  1572. turn_collector.agent_turns
  1573. if turn_collector is not None
  1574. else parsed.agent_turns
  1575. ),
  1576. )
  1577. return ExecutionEvidence(final_answer=final_answer, steps=parsed.steps, screenshot_paths=sorted(shots_dir.glob("*.png")), controller_exit_code=process.returncode if process.returncode is not None else -9, termination=termination, metrics=metrics)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.