Skip to content
Back to search

vercel

MarketplaceWorks with Claude CodeGitHub

Comprehensive Vercel ecosystem plugin — relational knowledge graph, skills for every major product, specialized agents, and Vercel conventions. Turns any AI agent into a Vercel expert.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/vercel/vercel-plugin with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/vercel/vercel-plugin'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/vercel-vercel-plugin/badge.svg)](/listings/vercel-vercel-plugin)

HTML

<a href="/listings/vercel-vercel-plugin"><img src="/api/listings/vercel-vercel-plugin/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Supported Tools

| Tool | Status | | ------------------------------------------------------------- | --------- | | [Claude Code](https://docs.anthropic.com/en/docs/claude-code) | Supported | | [Cursor](https://www.cursor.com) | Supported | | [GitHub Copilot CLI](https://github.com/github/copilot-cli) | Supported | | [OpenAI Codex](https://openai.com/index/codex/) | Supported | | [Grok Build](https://x.ai/news/grok-build-cli) | Supported | | [Kimi Code](https://kimi.com) | Supported |

Prerequisites

- One of the supported AI coding tools listed above - Node.js 18+ - [Bun](https://bun.sh)

Installation

npx plugins add vercel/vercel-plugin

That's it. The plugin installs Vercel context, skills, and a lightweight default hook profile.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 28 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "vercel",
  3. "owner": {
  4. "name": "Vercel"
  5. },
  6. "plugins": [
  7. {
  8. "name": "vercel-plugin",
  9. "source": "./",
  10. "description": "Build and deploy web apps and agents",
  11. "author": {
  12. "name": "Vercel"
  13. },
  14. "repository": "https://github.com/vercel/vercel-plugin",
  15. "license": "Apache-2.0",
  16. "keywords": [
  17. "vercel",
  18. "nextjs",
  19. "ai-sdk",
  20. "turborepo",
  21. "turbopack",
  22. "workflow",
  23. "deployment",
  24. "serverless"
  25. ]
  26. }
  27. ]
  28. }
README.md · README · 358 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # vercel-plugin
  2. ## Getting Started
  3. ### Supported Tools
  4. | Tool | Status |
  5. | ------------------------------------------------------------- | --------- |
  6. | [Claude Code](https://docs.anthropic.com/en/docs/claude-code) | Supported |
  7. | [Cursor](https://www.cursor.com) | Supported |
  8. | [GitHub Copilot CLI](https://github.com/github/copilot-cli) | Supported |
  9. | [OpenAI Codex](https://openai.com/index/codex/) | Supported |
  10. | [Grok Build](https://x.ai/news/grok-build-cli) | Supported |
  11. | [Kimi Code](https://kimi.com) | Supported |
  12. | [OpenCode](https://opencode.ai) 2 | Supported |
  13. ### Prerequisites
  14. - One of the supported AI coding tools listed above
  15. - Node.js 18+
  16. - [Bun](https://bun.sh)
  17. ### Installation
  18. ```bash
  19. npx plugins add vercel/vercel-plugin
  20. ```
  21. That's it. The plugin installs Vercel context, skills, and a lightweight default hook profile.
  22. For OpenCode 2, install it as a native OpenCode plugin:
  23. ```bash
  24. opencode plugin add github:vercel/vercel-plugin
  25. ```
  26. OpenCode loads the same skills, agents, commands (`/vercel:deploy`, `/vercel:env`, `/vercel:status`, `/vercel:bootstrap`), session context, and Vercel MCP server. Run `/mcps` in OpenCode to sign in to the MCP server. Node.js and Bun are not needed for OpenCode.
  27. ## What It Does
  28. This plugin gives AI agents a **relational knowledge graph** of the Vercel ecosystem plus a bundled skill library covering products, libraries, CLI, APIs, and workflows.
  29. ## How Do I Use This?
  30. After installing, the plugin keeps automatic behavior lightweight. Session-start activation now only kicks in for empty directories and detected Vercel, Next.js, or eve projects, and Vercel skills are no longer auto-injected on every tool call or every prompt by default. The default post-tool path is now observer-only. The skills remain available for direct use, and the repo still keeps the injection engine for targeted or future opt-in workflows.
  31. ## Vercel MCP connection
  32. The bundled `.mcp.json` connects to `https://mcp.vercel.com`. Authenticate and grant
  33. access to the intended teams before using account tools. MCP supports both reads
  34. and writes, including inspecting projects and logs, creating deployments, and
  35. updating resources. Inspect the tools and schemas exposed by your connection.
  36. Domain registration requires a quote and explicit approval; digital subscriptions,
  37. credits, and add-on purchases are unavailable through MCP.
  38. ## Components
  39. ### Ecosystem Graph (`vercel.md`)
  40. A text-form relational graph covering:
  41. - All Vercel products and their relationships
  42. - Decision matrices for choosing the right tool
  43. - Common cross-product workflows
  44. - Migration awareness for sunset products
  45. ### Skills (39 skills)
  46. | Skill | Covers |
  47. | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
  48. | `access-protected-vercel-deployment` | Authenticated CLI and browser access to protected preview and production deployments using `vc curl`, OIDC, and Trusted Sources |
  49. | `ai-gateway` | Unified model API, provider routing, failover, cost tracking, 100+ models |
  50. | `ai-sdk` | AI SDK 7 — text/object generation, streaming, tool calling, agents, MCP, providers, embeddings |
  51. | `auth` | Authentication integrations — Clerk, Better Auth, Descope, Auth0 setup for Next.js with Marketplace provisioning |
  52. | `bootstrap` | Project bootstrapping orchestrator — linking, env provisioning, db setup, first-run commands |
  53. | `botid` | Vercel BotID in code — `initBotId()` protect list, `checkBotId()` in handlers, verified bots, Deep Analysis, local testing |
  54. | `build-agents` | Default eve-first agent builder — create AI agents, agent apps, tools, channels, schedules, and Slack agents |
  55. | `cdn-caching` | Diagnose cache hit rate, stale content, revalidation behavior, per-request cache reasons, and ISR read/write cost across CDN/ISR/PPR |
  56. | `chat-sdk` | Multi-platform chat bots — Slack, Telegram, Teams, Discord, Google Chat, GitHub, Linear |
  57. | `create-a-backend` | Backend product and framework selection — Functions, Services, containers, Workflow, Queues, and databases |
  58. | `custom-metrics` | Emit numeric application and business measurements from Functions and query them with `vc metrics` |
  59. | `daemons` | Vercel Daemons — long-running Dockerfile-built workers declared under `daemons` in vercel.json, instances per region, SIGTERM, Queues poll mode |
  60. | `deployments-cicd` | Deployment and CI/CD — deploy, promote, rollback, --prebuilt, CI workflow files |
  61. | `domains` | Public domain search and pricing, CLI and Registrar API operations, project domains, DNS, transfers, and renewals |
  62. | `env-vars` | Environment variable management — .env files, vercel env commands, OIDC tokens |
  63. | `eve` | Filesystem-first framework for durable AI agents, agent applications, channels, sandboxes, schedules, evals, and frontend clients |
  64. | `flags-sdk` | Flags SDK code in Next.js and SvelteKit — `flag()` declarations, provider adapters, precompute, Flags Explorer endpoint |
  65. | `is-agentic` | Score how ready a website, domain, or public MCP endpoint is for AI agents, and act on the Is Agentic report |
  66. | `knowledge-update` | Knowledge update guidance for the plugin |
  67. | `marketplace` | Integration discovery and installation (`vercel integration add` / `vercel install`), provider agent skills, auto-provisioned env vars, unified billing |
  68. | `microfrontends` | Multi-zone and microfrontend architecture, routing, deployment boundaries, and shared dependencies |
  69. | `offload-to-sandbox` | Offload heavy builds, test suites, and parallel agent work to Vercel Sandboxes with the `vercel sandbox` CLI: base snapshots, sharding, cleanup |
  70. | `queues` | Vercel Queues — durable topics, `@vercel/queue` publish/consume, consumer groups, retries, delays, idempotency, Queues vs Workflows |
  71. | `react-best-practices` | React/Next.js performance optimization — 64 rules across 8 categories |
  72. | `routing-middleware` | Request interception before cache, rewrites, redirects, personalization — Edge/Node.js/Bun runtimes |
  73. | `runtime-cache` | Ephemeral per-region key-value cache, tag-based invalidation, shared across Functions/Middleware/Builds |
  74. | `vercel-agent` | Vercel Agent chat (dashboard, Slack), code review, investigations, approved actions, product installation |
  75. | `vercel-cli` | All CLI commands — deploy, env, dev, domains, cache management, MCP integration, marketplace |
  76. | `vercel-connect` | Managed OAuth tokens and third-party connections for apps, MCP servers, and eve agents |
  77. | `vercel-firewall` | DDoS protection, WAF rules, rate limiting, bot management (Bot Protection, AI Bots, bot name/category rules), and IP controls |
  78. | `vercel-flags` | Vercel Flags — plan flags, choose an integration, `vercel flags` CLI, rollouts, A/B tests, `@vercel/flags-core`, safe retirement |
  79. | `vercel-functions` | Node.js, Bun, Python, and Rust runtimes on Fluid Compute, streaming, WebSockets, legacy Cron Jobs, configuration |
  80. | `vercel-sandbox` | Ephemeral Firecracker microVMs for running untrusted/AI-generated code safely |
  81. | `vercel-schedules` | Vercel Schedules — static `schedules` in vercel.json, private `schedule/v1beta` functions, dynamic `@vercel/schedules`, one-time runs, Cron Jobs migration |
  82. | `vercel-services` | Multiple frontends and backends in one project, with public rewrites and private service bindings |
  83. | `vercel-storage` | Blob, Global Config (formerly Edge Config), Neon Postgres, Upstash Redis, migration from sunset packages |
  84. | `verification` | Full-story verification — infers user story, verifies end-to-end browser → API → data → response |
  85. | `workflow` | Workflow SDK — durable execution, WorkflowAgent, steps, Worlds, pause/resume |
  86. Next.js framework guidance comes from Next.js itself, following the [official agent setup](https://nextjs.org/docs/app/guides/ai-agents): version-matched docs bundled in `node_modules/next/dist/docs/` (with `AGENTS.md` generated by `next dev` on 16.3+), plus workflow skills installed with `npx skills add vercel/next.js`. The plugin points agents there instead of shipping its own Next.js skills.
  87. ### Agents (3 specialists)
  88. | Agent | Expertise |
  89. | ----------------------- | ------------------------------------------------------------------------------- |
  90. | `deployment-expert` | CI/CD pipelines, deploy strategies, troubleshooting, environment variables |
  91. | `performance-optimizer` | Core Web Vitals, rendering strategies, caching, asset optimization |
  92. | `ai-architect` | AI application design, model selection, streaming architecture, MCP integration |
  93. ### Commands (4 commands)
  94. | Command | Purpose |
  95. | -------------------------- | ------------------------------------------------------- |
  96. | `/vercel-plugin:bootstrap` | Bootstrap project — linking, env provisioning, db setup |
  97. | `/vercel-plugin:deploy` | Deploy to Vercel (preview or production) |
  98. | `/vercel-plugin:env` | Manage environment variables |
  99. | `/vercel-plugin:status` | Project status overview |
  100. ### Hooks
  101. Lifecycle hooks that run automatically during your session:
  102. - **Session start context injection** — Injects a thin Vercel session context plus the knowledge-update guidance for empty directories and detected Vercel, Next.js, or eve projects
  103. - **Session start repo profiler** — Scans config files and dependencies to set likely-skill hints, but only after that same activation check passes
  104. ## Usage
  105. After installing, session context is injected automatically only for empty directories and detected Vercel, Next.js, or eve projects. Vercel skills are available on demand, and you can invoke them directly via slash commands:
  106. ```
  107. /vercel-plugin:build-agents
  108. /vercel-plugin:ai-sdk
  109. /vercel-plugin:deploy prod
  110. ```
  111. ## Telemetry
  112. Telemetry is on by default and can be disabled with `VERCEL_PLUGIN_TELEMETRY=off`.
  113. What is collected:
  114. - `dau:active_today`: sent at most once per UTC day when the plugin runs.
  115. - `plugin:first_use`: sent once per local user profile the first time the plugin successfully reports telemetry.
  116. - `plugin:version`: sent with telemetry batches so usage can be grouped by plugin version.
  117. - `plugin:install_id`: the locally stored random installation UUID.
  118. - `plugin:agent_harness`: each distinct detected agent harness category observed per installation per UTC day.
  119. - `skill:invoked`: the name of a skill or command that ships with this plugin (for example `ai-sdk`, `workflow`, `deploy`) each time it is loaded through the agent's `Skill` tool — whether you typed the slash command or the model chose it. See [Skill telemetry](#skill-telemetry) below.
  120. - `skill:injected`: the same skill names, sent when the plugin's opt-in injection hooks (`pretooluse-skill-inject`, `user-prompt-submit-skill-inject`) add a skill to the agent's context. These hooks are not registered by default, so most installs never emit this key. The key therefore records *how* a skill reached the agent; the value records *which* skill.
  121. Each telemetry event contains only:
  122. - `id`: a random event UUID.
  123. - `event_time`: the event timestamp.
  124. - `key`: one of the event names listed above.
  125. - `value`: `"1"` for counters, the plugin version, the random installation UUID, the detected harness, or the plugin skill name, depending on the event key.
  126. The request also sends HTTP headers used by the telemetry bridge:
  127. - `x-vercel-plugin-topic-id`: `dau` for the daily ping, `generic` for skill events.
  128. - `x-vercel-plugin-session-id`: for the daily ping, a random UUID generated for that telemetry request; for skill events, a random UUID the plugin mints once per agent session (see below).
  129. - `x-vercel-plugin-version`: the plugin version embedded at build time.
  130. The installation ID is generated on the first telemetry-enabled plugin session and reused for that local installation. It is not derived from device, account, project, or user information. The harness value identifies Claude Code (including Claude Cowork), Cursor, Codex, GitHub Copilot, Kimi Code, or Grok using [`detect-agent`](https://github.com/vercel/detect-agent). The OpenCode plugin always reports `opencode`. A detected but unsupported or custom harness is reported as `other`; `unknown` means no harness was detected. Raw custom harness names are never sent.
  131. Prompt text, bash commands, tool-call contents, skill arguments, file paths, project names, account IDs, harness versions, and skill-injection details are not collected.
  132. ### Skill telemetry
  133. The plugin reports which of *its own* skills get used so we can see which guidance is valuable and which is not. It is deliberately narrow:
  134. - **Only this plugin's skills are reported.** When the agent loads a skill (for example `/vercel:ai-sdk`), a `PostToolUse` hook on the `Skill` tool requires the namespace to be this plugin's (`vercel` or `vercel-plugin`) *and* the bare name to exist in the `skills/` or `commands/` directories that ship with it. Anything else — un-namespaced personal skills (`deploy`), other plugins' skills even with the same name (`other-plugin:deploy`), typos — is dropped entirely; nothing is sent, not even an "other" bucket.
  135. - **Only the skill name is sent.** Skill arguments, the tool's response, the prompt that triggered the skill, and anything else in the tool call are never read past the name check and never leave your machine.
  136. - **No harness identifiers.** Events from one agent session share a random UUID the plugin mints itself and stores in a session-scoped temp file (`<tmpdir>/vercel-plugin-<session>-telemetry-session-id.txt`, removed at session end). Your agent's own session ID is never sent, so skill usage cannot be joined to any other telemetry the agent produces.
  137. - **Harness category only.** The session-start hook records the detected harness category (`claude-code`, `cursor`, `codex`, `github-copilot`, `kimi`, `grok`, `opencode`, `other`, or `unknown` — the same values as `plugin:agent_harness`) in a session temp file (`<tmpdir>/vercel-plugin-<session>-agent-harness.txt`), and skill events carry it so usage can be broken down per harness. No harness version or raw agent name is sent.
  138. - **Coverage by harness.** The `Skill` tool is Claude Code's contract, and Cursor's Claude-compatible hook bridge is accepted too (`conversation_id` payloads). Cursor's own hook API does not expose a skill-load event or a `Skill` tool name, so skills loaded natively by Cursor are not currently observable and are simply not counted.
  139. - **OpenCode.** OpenCode skills have no plugin namespace, so the OpenCode plugin reports a load of OpenCode's `skill` tool only when the skill ID is one this plugin ships *and* the loaded file is the plugin's own copy, not a same-ID skill of yours. Commands do not go through the `skill` tool in OpenCode, so they are not counted. The per-session UUID lives in memory instead of a temp file, and subagent sessions share their lead session's UUID.
  140. - **It never slows the agent down.** The hook validates the name and exits immediately (a few milliseconds); the network request runs in a detached background process with a 3-second timeout. If the bridge is unreachable the event is simply lost — there is no retry queue and nothing is persisted.
  141. - **Same off switch.** `VERCEL_PLUGIN_TELEMETRY=off` disables it along with everything else.
  142. Each skill event request also carries `plugin:version`, `plugin:install_id`, and (when known) `plugin:agent_harness` so usage can be grouped by version, installation, and harness, exactly as the daily ping does.
  143. How it is tracked:
  144. - Events are sent to Vercel's public telemetry bridge at `https://telemetry.vercel.com/api/vercel-plugin/v1/events`.
  145. - The bridge only forwards events from plugin versions `0.40.0` and newer.
  146. - Local throttle files are stored under `~/.config/vercel-plugin/`:
  147. - `dau-stamp` prevents sending `dau:active_today` more than once per UTC day.
  148. - `harness-stamp-<harness>` prevents sending the same `plugin:agent_harness` value more than once per UTC day.
  149. - `first-use-stamp` prevents sending `plugin:first_use` more than once.
  150. - `installation-id` stores the random installation UUID. It is used only by plugin telemetry and is not written to `active-session.json`.
  151. - Stamp files are written only after the telemetry bridge returns a successful response, so failed sends can retry later.
  152. - Skill events are not throttled or stamped; each load of a plugin skill is one event, sent once and never retried.
  153. - `active-session.json` is refreshed on session start with the plugin version and expiry timestamp. It lets Vercel CLI telemetry identify commands run while a recent Vercel plugin session marker is present. It contains no prompt text, file paths, project names, account IDs, tool-call contents, or skill-injection details.
  154. Behavior:
  155. - Unset `VERCEL_PLUGIN_TELEMETRY`: telemetry is enabled.
  156. - `VERCEL_PLUGIN_TELEMETRY=off`: disables all telemetry, including `dau:active_today`, `plugin:first_use`, `skill:invoked`, and `skill:injected`, and does not create an installation ID if one does not already exist.
  157. Where to set `VERCEL_PLUGIN_TELEMETRY`:
  158. - OpenCode: its background service keeps the environment it started with, so run `opencode service restart` after setting the variable. You can also disable telemetry in `opencode.json` with `"plugins": [{ "package": "github:vercel/vercel-plugin", "options": { "telemetry": false } }]`.
  159. - macOS / Linux: add it to the shell profile for the environment that launches your agent, such as `~/.zshrc`, `~/.bashrc`, `~/.bash_profile`, or `~/.config/fish/config.fish`, then restart that terminal or app session.
  160. - Windows: set it in the PowerShell environment that launches your agent, add it to your PowerShell profile (`$PROFILE`), or set it as a persistent user environment variable.
  161. Examples:
  162. ```bash
  163. export VERCEL_PLUGIN_TELEMETRY=off
  164. ```
  165. ```powershell
  166. setx VERCEL_PLUGIN_TELEMETRY off
  167. ```
  168. ## Upstream Skill Sync
  169. 9 skills are synced from their upstream source repos on [skills.sh](https://skills.sh). Each synced skill uses an **overlay + upstream** model:
  170. ```
  171. skills/<name>/
  172. ├── overlay.yaml # Plugin metadata (priority, pathPatterns, validate, chainTo)
  173. ├── upstream/
  174. │ ├── SKILL.md # Pure upstream file (synced from source repo)
  175. │ └── references/ # Upstream reference files
  176. └── SKILL.md # Build output: overlay + upstream body (auto-generated)
  177. ```
  178. - `overlay.yaml` is ours — injection metadata that drives the hook system. Never overwritten by sync.
  179. - `upstream/SKILL.md` is theirs — pulled from the source repo, never manually edited.
  180. - `SKILL.md` is the build output — auto-generated by `bun run build:skills`.
  181. ### Synced Skills
  182. | Skill | Upstream Repo |
  183. | ----------------------- | ------------------------------------------------------------------------- |
  184. | `ai-sdk` | [vercel/ai](https://github.com/vercel/ai) |
  185. | `chat-sdk` | [vercel/chat](https://github.com/vercel/chat) |
  186. | `eve` | [vercel/eve](https://github.com/vercel/eve) |
  187. | `flags-sdk` | [vercel/flags](https://github.com/vercel/flags) |
  188. | `is-agentic` | vercel-labs/is-agentic (private); public copy at [is-agentic.com](https://is-agentic.com/.well-known/agent-skills/is-agentic/SKILL.md) |
  189. | `react-best-practices` | [vercel-labs/agent-skills](https://github.com/vercel-labs/agent-skills) |
  190. | `vercel-cli` | [vercel/vercel](https://github.com/vercel/vercel) |
  191. | `vercel-flags` | [vercel/flags](https://github.com/vercel/flags) |
  192. | `vercel-sandbox` | [vercel-labs/agent-browser](https://github.com/vercel-labs/agent-browser) |
  193. | `workflow` | [vercel/workflow](https://github.com/vercel/workflow) |
  194. ### Syncing an Upstream Skill
  195. Drop the updated file into `upstream/` and rebuild:
  196. ```bash
  197. cp /path/to/new/SKILL.md skills/ai-sdk/upstream/SKILL.md
  198. bun run build:skills
  199. ```
  200. ### CI Check
  201. ```bash
  202. bun run build:skills:check
  203. ```
  204. Exits non-zero if any `SKILL.md` is stale. Add to CI to catch drift.
  205. ## Architecture
  206. ```
  207. vercel-plugin/
  208. ├── .plugin/plugin.json # Plugin manifest
  209. ├── vercel.md # Ecosystem graph + conventions (injected via SessionStart hook)
  210. ├── skills/ # 39 skills
  211. │ ├── ai-sdk/ # Upstream-synced skill example:
  212. │ │ ├── overlay.yaml # Plugin injection metadata
  213. │ │ ├── upstream/ # Pure upstream content
  214. │ │ │ ├── SKILL.md
  215. │ │ │ └── references/ # Only when the upstream repo ships them
  216. │ │ ├── SKILL.md # Build output (overlay + upstream)
  217. │ │ └── references/ # Copied from upstream at build time (when present)
  218. │ ├── bootstrap/ # Plugin-only skill example:
  219. │ │ └── SKILL.md # Entirely ours
  220. │ └── ...
  221. ├── agents/ # 3 specialist agents
  222. ├── commands/ # 4 slash commands
  223. ├── scripts/
  224. │ ├── build-skills.ts # Rules engine: overlay + upstream → SKILL.md
  225. │ └── build-from-skills.ts # Resolves {{include:skill:...}} in templates
  226. └── hooks/ # SessionStart injection, repo profiler, skill injection, deprecation guard
  227. ├── opencode-plugin.mjs # OpenCode 2 entry (package.json `exports`)
  228. └── src/ # TypeScript source (compiled to .mjs via tsup)
  229. ```
  230. ## Build Pipeline
  231. ```bash
  232. bun run build:all # Runs all 3 stages in order
  233. bun run build:skills # Stage 1: Merge overlay + upstream → SKILL.md
  234. bun run build:hooks # Stage 2: Compile hook TypeScript → .mjs
  235. bun run build:from-skills # Stage 3: Resolve template includes
  236. ```
  237. Hooks and developer tools read skill metadata directly from `skills/*/SKILL.md`.
  238. Builds only produce the skill files, hook scripts, and agent/command instructions
  239. used by the plugin. Validation checks the source files without writing reports
  240. back into the repository.
  241. ## Ecosystem Coverage (September 2026)
  242. - Next.js 16 (App Router, Cache Components, Proxy, View Transitions)
  243. - AI SDK 7 (Agents, WorkflowAgent, harnesses, MCP, DevTools, telemetry)
  244. - AI Elements (pre-built React components for AI interfaces)
  245. - Chat SDK (multi-platform chat bots — Slack, Telegram, Teams, Discord)
  246. - Workflow SDK (WorkflowAgent, Worlds, open source)
  247. - AI Gateway (100+ models, provider routing, cost tracking)
  248. - Vercel Functions (Fluid Compute, Node.js/Bun/Python/Rust, streaming, WebSockets, legacy Cron Jobs)
  249. - Vercel Services (multiple frontends and backends, public rewrites, private bindings)
  250. - Storage (Blob, Global Config, Neon Postgres, Upstash Redis)
  251. - Routing Middleware (request interception, Edge/Node.js/Bun runtimes)
  252. - Runtime Cache API (per-region KV cache, tag-based invalidation)
  253. - Vercel Flags (feature flags, Flags Explorer, gradual rollouts, A/B testing)
  254. - Vercel Queues (durable event streaming, topics, consumer groups, retries)
  255. - Vercel Daemons (long-running background workers from a Dockerfile, instances per region)
  256. - Vercel Schedules (recurring and one-time runs of private functions, routes, or queue topics; replaces Cron Jobs)
  257. - Vercel Agent (chat, code review, investigations, approved actions)
  258. - Vercel Sandbox (Firecracker microVMs for untrusted code)
  259. - Sign in with Vercel (OAuth 2.0/OIDC identity provider)
  260. - Auth integrations (Clerk, Descope, Auth0)
  261. - Turborepo (--affected, remote caching, Rust core)
  262. - Turbopack (default bundler in Next.js 16)
  263. - v0 (agentic intelligence, GitHub integration)
  264. - Vercel CLI (cache management, MCP integration, marketplace discovery)
  265. - Protected deployment access (`vc curl`, local OIDC tokens, Trusted Sources)
  266. - Vercel Firewall and BotID (WAF rules, managed bot rules, rate limiting, per-route bot checks)
  267. - Vercel Observability (Analytics, Speed Insights, Drains)
  268. - Vercel Marketplace (one-click integrations, unified billing)
  269. - Agent Browser (browser automation for dev server verification and testing)
  270. ## Reporting Issues
  271. If something doesn't work right, a skill gives bad advice, or injection doesn't fire when it should — file an issue on [GitHub](https://github.com/vercel/vercel-plugin/issues). Include:
  272. - What you were building
  273. - What the plugin injected (or didn't) — enable debug logs with `VERCEL_PLUGIN_LOG_LEVEL=debug`
  274. - What was wrong about it
  275. ## License
  276. [Apache-2.0](LICENSE)
.claude/skills/benchmark-sandbox/sandbox-runner.ts · Bundled script · 848 lines · 2 findings marked
  1. #!/usr/bin/env bun
  2. /**
  3. * Sandbox benchmark runner: provisions Vercel Sandboxes from a snapshot,
  4. * runs benchmark scenarios with Claude Code + plugin, and extracts artifacts.
  5. *
  6. * Scenarios are imported from scripts/benchmark-runner.ts (no duplication).
  7. *
  8. * Usage:
  9. * bun run .claude/skills/benchmark-sandbox/sandbox-runner.ts [options]
  10. *
  11. * --snapshot-only Create/update the base snapshot and exit
  12. * --quick Run Tier 1 only (scenarios 01, 04, 09)
  13. * --scenario <slug> Run a single scenario by slug
  14. * --concurrency <n> Max parallel sandboxes (default: 3)
  15. * --timeout <ms> Per-scenario timeout (default: 300000 = 5 min)
  16. * --results-dir <p> Override results directory
  17. * --help Print usage and exit
  18. */
  19. import { Sandbox } from "@vercel/sandbox";
  20. import { mkdir, writeFile, readFile, readdir, stat, unlink } from "node:fs/promises";
  21. import { join, resolve } from "node:path";
  22. import { parseArgs } from "node:util";
  23. import { homedir } from "node:os";
  24. import { execSync } from "node:child_process";
  25. import type { RunSummary, ScenarioSummary, SnapshotMeta } from "./types.js";
  26. import {
  27. RunnerError,
  28. scenarioStatus,
  29. overallStatus,
  30. EXIT_ALL_PASS,
  31. EXIT_SOME_FAIL,
  32. EXIT_FATAL,
  33. } from "./types.js";
  34. import { Logger, type LogFormat } from "./logger.js";
  35. // ---------------------------------------------------------------------------
  36. // Import scenarios from the canonical source
  37. // ---------------------------------------------------------------------------
  38. import { PROJECTS, type BenchmarkProject } from "../../../scripts/benchmark-scenarios.js";
  39. const SCENARIOS: BenchmarkProject[] = PROJECTS;
  40. // Tier 1 quick-run slugs
  41. const TIER1_SLUGS = new Set(["01-doc-qa-agent", "04-multi-model-router", "09-code-sandbox-tutor"]);
  42. // ---------------------------------------------------------------------------
  43. // CLI
  44. // ---------------------------------------------------------------------------
  45. const { values: flags } = parseArgs({
  46. options: {
  47. "snapshot-only": { type: "boolean", default: false },
  48. "force-snapshot": { type: "boolean", default: false },
  49. quick: { type: "boolean", default: false },
  50. scenario: { type: "string" },
  51. concurrency: { type: "string", default: "3" },
  52. timeout: { type: "string", default: "300000" },
  53. "results-dir": { type: "string" },
  54. json: { type: "boolean", default: false },
  55. "log-format": { type: "string", default: "human" },
  56. help: { type: "boolean", default: false },
  57. },
  58. strict: true,
  59. });
  60. if (flags.help) {
  61. console.log(`Usage: bun run .claude/skills/benchmark-sandbox/sandbox-runner.ts [options]
  62. --snapshot-only Create/update the base snapshot and exit
  63. --force-snapshot Force fresh snapshot creation (ignore cache)
  64. --quick Run Tier 1 only (scenarios 01, 04, 09)
  65. --scenario <slug> Run a single scenario by slug
  66. --concurrency <n> Max parallel sandboxes (default: 3, max: 10)
  67. --timeout <ms> Per-scenario timeout (default: 300000 = 5 min)
  68. --results-dir <p> Override results directory
  69. --json Write only RunSummary JSON to stdout (all other output → stderr)
  70. --log-format <fmt> Log format: human (default) or json (NDJSON events to stderr)
  71. --help Print usage
  72. Exit codes:
  73. 0 All scenarios passed
  74. 1 One or more scenarios failed or timed out
  75. 2 Fatal runner error (snapshot failure, scenario load failure, etc.)`);
  76. process.exit(0);
  77. }
  78. const JSON_MODE = flags.json!;
  79. const LOG_FORMAT = (flags["log-format"] === "json" ? "json" : "human") as LogFormat;
  80. // runId declared early so Logger can use it; the timestamped form is set later in main()
  81. let runId = `run-${new Date().toISOString().replace(/[:.]/g, "-").slice(0, 19)}`;
  82. const logger = new Logger({ format: LOG_FORMAT, runId });
  83. /** Shorthand — all human output goes to stderr (stdout reserved for --json RunSummary). */
  84. const log = (msg: string) => logger.info(msg);
  85. const warn = (msg: string) => logger.warn(msg);
  86. const MAX_CONCURRENCY = 10;
  87. const WARN_CONCURRENCY = 6;
  88. const CONCURRENCY_RAW = parseInt(flags.concurrency!, 10);
  89. if (CONCURRENCY_RAW > MAX_CONCURRENCY) {
  90. logger.error(`Concurrency ${CONCURRENCY_RAW} exceeds maximum of ${MAX_CONCURRENCY}. Clamping to ${MAX_CONCURRENCY}.`);
  91. }
  92. if (CONCURRENCY_RAW > WARN_CONCURRENCY && CONCURRENCY_RAW <= MAX_CONCURRENCY) {
  93. warn(`Warning: concurrency ${CONCURRENCY_RAW} exceeds ${WARN_CONCURRENCY} — may hit sandbox rate limits.`);
  94. }
  95. const CONCURRENCY = Math.min(Math.max(CONCURRENCY_RAW, 1), MAX_CONCURRENCY);
  96. const TIMEOUT_MS = parseInt(flags.timeout!, 10);
  97. const LOCAL_PLUGIN_DIR = join(homedir(), "dev", "vercel-plugin");
  98. const SANDBOX_HOME = "/home/vercel-sandbox";
  99. const SANDBOX_PLUGIN_DIR = `${SANDBOX_HOME}/vercel-plugin`;
  100. const DEFAULT_RESULTS = join(homedir(), "dev", "vercel-plugin-testing", "sandbox-results");
  101. const RESULTS_DIR = resolve(flags["results-dir"] ?? DEFAULT_RESULTS);
  102. const SNAPSHOT_CACHE_PATH = join(DEFAULT_RESULTS, ".snapshot-cache.json");
  103. const SNAPSHOT_MAX_AGE_MS = 24 * 60 * 60 * 1000; // 24 hours
  104. // ---------------------------------------------------------------------------
  105. // Helpers
  106. // ---------------------------------------------------------------------------
  107. function elapsed(start: number): string {
  108. return `${((performance.now() - start) / 1000).toFixed(1)}s`;
  109. }
  110. function resolveApiKey(): string {
  111. // VERCEL_API_KEY is for sandbox provisioning, not for Claude Code
  112. const key =
  113. process.env.ANTHROPIC_API_KEY ??
  114. process.env.AI_GATEWAY_API_KEY;
  115. if (key) return key;
  116. // Try macOS Keychain via apiKeyHelper
  117. try {
  118. const keychainKey = execSync(
  119. 'security find-generic-password -a "$USER" -s "ANTHROPIC_AUTH_TOKEN" -w',
  120. { encoding: "utf-8", timeout: 5000 },
  121. ).trim();
  122. if (keychainKey) return keychainKey;
  123. } catch { /* keychain not available or key not found */ }
  124. logger.error("Missing API key. Set ANTHROPIC_API_KEY or AI_GATEWAY_API_KEY (or store in macOS Keychain as ANTHROPIC_AUTH_TOKEN)");
  125. process.exit(1);
  126. }
  127. function resolveBaseUrl(): string | undefined {
  128. return process.env.ANTHROPIC_BASE_URL ?? "https://ai-gateway.vercel.sh";
  129. }
  130. /** Essential plugin directories/files to upload into the sandbox. */
  131. const PLUGIN_UPLOAD_DIRS = ["hooks", "skills", "generated"];
  132. const PLUGIN_UPLOAD_FILES = ["hooks/hooks.json", "package.json"];
  133. /**
  134. * Recursively collect files from a local directory, returning
  135. * { relativePath: fileContent } pairs suitable for sandbox.writeFiles().
  136. */
  137. async function collectPluginFiles(): Promise<Record<string, string>> {
  138. const files: Record<string, string> = {};
  139. async function walkDir(dir: string, prefix: string): Promise<void> {
  140. const entries = await readdir(join(LOCAL_PLUGIN_DIR, dir), { withFileTypes: true });
  141. for (const entry of entries) {
  142. const relPath = join(dir, entry.name);
  143. const fullPath = join(LOCAL_PLUGIN_DIR, relPath);
  144. if (entry.isDirectory()) {
  145. // Skip node_modules, .git, src (only need compiled hooks)
  146. if (["node_modules", ".git", "src", ".claude", "tests", "scripts", ".playground"].includes(entry.name)) continue;
  147. await walkDir(relPath, prefix);
  148. } else if (entry.isFile()) {
  149. // Skip .mts source files (only need compiled .mjs), test files, and large files
  150. if (entry.name.endsWith(".mts") || entry.name.endsWith(".test.ts")) continue;
  151. const s = await stat(fullPath);
  152. if (s.size > 200_000) continue; // skip files > 200KB
  153. const content = await readFile(fullPath, "utf-8");
  154. files[join(prefix, relPath)] = content;
  155. }
  156. }
  157. }
  158. for (const dir of PLUGIN_UPLOAD_DIRS) {
  159. await walkDir(dir, "");
  160. }
  161. // Also upload root-level files
  162. for (const f of PLUGIN_UPLOAD_FILES) {
  163. try {
  164. const content = await readFile(join(LOCAL_PLUGIN_DIR, f), "utf-8");
  165. files[f] = content;
  166. } catch { /* optional file */ }
  167. }
  168. return files;
  169. }
  170. /**
  171. * Upload the local plugin into the sandbox at SANDBOX_PLUGIN_DIR,
  172. * then run `npx add-plugin <sandbox-path>` to install it.
  173. */
  174. async function uploadAndInstallPlugin(
  175. sandbox: SandboxInstance,
  176. projectDir: string,
  177. ): Promise<{ out: string; err: string; exit: number }> {
  178. const pluginFiles = await collectPluginFiles();
  179. const entries = Object.entries(pluginFiles);
  180. log(` [plugin] Uploading ${entries.length} files to sandbox via writeFiles()...`);
  181. // Use native sandbox.writeFiles() API — much more reliable than heredocs
  182. const fileBatch = entries.map(([relPath, content]) => ({
  183. path: join(SANDBOX_PLUGIN_DIR, relPath),
  184. content: Buffer.from(content, "utf-8"),
  185. }));
  186. // Upload in chunks of 10 to avoid payload size limits
  187. const CHUNK_SIZE = 10;
  188. const totalChunks = Math.ceil(fileBatch.length / CHUNK_SIZE);
  189. logger.event("plugin.upload.started", { file_count: fileBatch.length, chunk_size: CHUNK_SIZE, total_chunks: totalChunks });
  190. for (let i = 0; i < fileBatch.length; i += CHUNK_SIZE) {
  191. const chunk = fileBatch.slice(i, i + CHUNK_SIZE);
  192. const chunkIndex = Math.floor(i / CHUNK_SIZE);
  193. const chunkBytes = chunk.reduce((a, f) => a + f.content.length, 0);
  194. try {
  195. await sandbox.writeFiles(chunk);
  196. logger.event("plugin.upload.chunk", { index: chunkIndex, total: totalChunks, bytes: chunkBytes });
  197. } catch (err: any) {
  198. logger.error(` [plugin] writeFiles chunk ${i}-${i + chunk.length} failed: ${err.message}`);
  199. logger.error(` [plugin] Paths: ${chunk.map(f => f.path).join(", ")}`);
  200. logger.error(` [plugin] Total bytes: ${chunkBytes}`);
  201. throw err;
  202. }
  203. }
  204. logger.event("plugin.upload.completed", { file_count: fileBatch.length });
  205. // Manually write Claude Code plugin config instead of using add-plugin
  206. // (add-plugin calls `claude mcp add-json` which requires claude in /bin/sh PATH)
  207. const globalSettings = {
  208. extraKnownMarketplaces: {
  209. "vercel-plugin": {
  210. source: { source: "directory", path: SANDBOX_PLUGIN_DIR },
  211. },
  212. },
  213. enabledPlugins: { "vercel-plugin@vercel-plugin": true },
  214. };
  215. const projectSettings = {
  216. enabledPlugins: { "vercel-plugin@vercel-plugin": true },
  217. };
  218. log(` [plugin] Writing plugin config manually...`);
  219. await sandbox.writeFiles([
  220. {
  221. path: `${SANDBOX_HOME}/.claude/settings.json`,
  222. content: Buffer.from(JSON.stringify(globalSettings, null, 2)),
  223. },
  224. {
  225. path: `${projectDir}/.claude/settings.json`,
  226. content: Buffer.from(JSON.stringify(projectSettings, null, 2)),
  227. },
  228. ]);
  229. return { out: "Plugin config written manually", err: "", exit: 0 };
  230. }
  231. type SandboxInstance = InstanceType<typeof Sandbox>;
  232. async function run(
  233. sandbox: SandboxInstance,
  234. cmd: string,
  235. args: string[],
  236. opts?: { timeout?: number },
  237. ): Promise<{ out: string; err: string; exit: number }> {
  238. // @vercel/sandbox runCommand does not accept `timeout` — use AbortSignal instead
  239. const runOpts: { signal?: AbortSignal } = {};
  240. if (opts?.timeout) {
  241. runOpts.signal = AbortSignal.timeout(opts.timeout);
  242. }
  243. try {
  244. const result = await sandbox.runCommand(cmd, args, runOpts);
  245. return {
  246. out: (await result.stdout()).trim(),
  247. err: (await result.stderr()).trim(),
  248. exit: (result as any).exitCode ?? 0,
  249. };
  250. } catch (err: any) {
  251. // runCommand throws on 400 (e.g. executable_not_found) instead of returning exit code
  252. const msg = err.text ?? err.message ?? String(err);
  253. logger.error(` [run] Command failed: ${cmd} ${args.slice(0, 2).join(" ")}`);
  254. logger.error(` [run] Error: ${msg.slice(0, 300)}`);
  255. return { out: "", err: msg, exit: 127 };
  256. }
  257. }
  258. // ---------------------------------------------------------------------------
  259. // Snapshot cache
  260. // ---------------------------------------------------------------------------
  261. interface SnapshotCache {
  262. snapshotId: string;
  263. createdAt: string; // ISO timestamp
  264. }
  265. async function loadCachedSnapshot(): Promise<string | null> {
  266. try {
  267. const raw = await readFile(SNAPSHOT_CACHE_PATH, "utf-8");
  268. const cache: SnapshotCache = JSON.parse(raw);
  269. const age = Date.now() - new Date(cache.createdAt).getTime();
  270. if (age < SNAPSHOT_MAX_AGE_MS) {
  271. log(`[snapshot] Using cached snapshot ${cache.snapshotId} (age: ${(age / 3600_000).toFixed(1)}h)`);
  272. return cache.snapshotId;
  273. }
  274. log(`[snapshot] Cached snapshot expired (age: ${(age / 3600_000).toFixed(1)}h > 24h)`);
  275. return null;
  276. } catch {
  277. return null; // no cache file or invalid JSON
  278. }
  279. }
  280. async function saveCachedSnapshot(snapshotId: string): Promise<void> {
  281. await mkdir(join(SNAPSHOT_CACHE_PATH, ".."), { recursive: true });
  282. const cache: SnapshotCache = { snapshotId, createdAt: new Date().toISOString() };
  283. await writeFile(SNAPSHOT_CACHE_PATH, JSON.stringify(cache, null, 2));
  284. }
  285. async function removeCachedSnapshot(): Promise<void> {
  286. try {
  287. await unlink(SNAPSHOT_CACHE_PATH);
  288. } catch { /* no cache to remove */ }
  289. }
  290. // ---------------------------------------------------------------------------
  291. // Snapshot verification — boot from snapshot, run `claude --version`
  292. // ---------------------------------------------------------------------------
  293. async function verifySnapshot(snapshotId: string): Promise<boolean> {
  294. logger.event("snapshot.verify.started", { snapshot_id: snapshotId });
  295. let sandbox: SandboxInstance | undefined;
  296. try {
  297. sandbox = await Sandbox.create({ fromSnapshot: snapshotId, timeout: 120_000 });
  298. const ver = await run(sandbox, "sh", ["-c", "claude --version"], { timeout: 30_000 });
  299. if (ver.exit === 0 && ver.out.length > 0) {
  300. logger.event("snapshot.verify.succeeded", { snapshot_id: snapshotId, claude_version: ver.out.slice(0, 80) });
  301. return true;
  302. }
  303. logger.event("snapshot.verify.failed", { snapshot_id: snapshotId, exit: ver.exit, stderr: ver.err.slice(0, 200) });
  304. return false;
  305. } catch (err: any) {
  306. logger.event("snapshot.verify.failed", { snapshot_id: snapshotId, error: err.message?.slice(0, 200) ?? String(err) });
  307. return false;
  308. } finally {
  309. if (sandbox) {
  310. try { await sandbox.stop(); } catch { /* already stopped */ }
  311. }
  312. }
  313. }
  314. // ---------------------------------------------------------------------------
  315. // Snapshot creation
  316. // ---------------------------------------------------------------------------
  317. async function createSnapshot(): Promise<string> {
  318. const t0 = performance.now();
  319. const apiKey = resolveApiKey();
  320. const baseUrl = resolveBaseUrl();
  321. log("[snapshot] Creating base sandbox (node24)...");
  322. const env: Record<string, string> = {
  323. ANTHROPIC_API_KEY: apiKey,
  324. VERCEL_PLUGIN_LOG_LEVEL: "trace",
  325. };
  326. if (baseUrl) env.ANTHROPIC_BASE_URL = baseUrl;
  327. const sandbox = await Sandbox.create({ runtime: "node24", env, timeout: 900_000 });
  328. log(`[snapshot] Sandbox ${sandbox.sandboxId} created (${elapsed(t0)})`);
  329. // Install Claude Code
  330. log("[snapshot] Installing Claude Code...");
  331. const t1 = performance.now();
  332. const install = await run(sandbox, "sh", ["-c", "npm install -g @anthropic-ai/claude-code"]);
  333. if (install.exit !== 0) {
  334. await sandbox.stop();
  335. throw new Error(`Claude Code install failed: ${install.err.slice(0, 300)}`);
  336. }
  337. log(`[snapshot] Claude Code installed (${elapsed(t1)})`);
  338. // Symlink claude + node binaries to /usr/local/bin so /bin/sh subprocesses can find them
  339. await run(sandbox, "sh", [
  340. "-c",
  341. "ln -sf $(which claude) /usr/local/bin/claude && ln -sf $(which node) /usr/local/bin/node && ln -sf $(which npm) /usr/local/bin/npm && ln -sf $(which npx) /usr/local/bin/npx",
  342. ]);
  343. // Verify claude --version (use sh -c to ensure PATH resolution)
  344. const ver = await run(sandbox, "sh", ["-c", "claude --version"]);
  345. log(`[snapshot] claude version: ${ver.out}`);
  346. // Take snapshot
  347. log("[snapshot] Taking snapshot...");
  348. const t2 = performance.now();
  349. const snapshot = await sandbox.snapshot();
  350. const snapshotId = (snapshot as any).snapshotId ?? (snapshot as any).id ?? String(snapshot);
  351. log(`[snapshot] Snapshot created: ${snapshotId} (${elapsed(t2)})`);
  352. await sandbox.stop();
  353. log(`[snapshot] Total: ${elapsed(t0)}`);
  354. return snapshotId;
  355. }
  356. // ---------------------------------------------------------------------------
  357. // Per-scenario runner
  358. // ---------------------------------------------------------------------------
  359. interface ScenarioResult {
  360. slug: string;
  361. sandboxId: string;
  362. success: boolean;
  363. timedOut: boolean;
  364. durationMs: number;
  365. sessionMethod: string;
  366. expectedSkills: string[];
  367. claimedSkills: string[];
  368. hookEvidence: {
  369. claimDirs: boolean;
  370. seenFile: boolean;
  371. debugLogCount: number;
  372. preToolUseInStderr: boolean;
  373. userPromptInStderr: boolean;
  374. };
  375. error?: string;
  376. }
  377. async function runScenario(
  378. project: BenchmarkProject,
  379. snapshotId: string,
  380. apiKey: string,
  381. baseUrl: string | undefined,
  382. ): Promise<ScenarioResult> {
  383. const t0 = performance.now();
  384. const projectDir = `${SANDBOX_HOME}/${project.slug}`;
  385. const env: Record<string, string> = {
  386. ANTHROPIC_API_KEY: apiKey,
  387. VERCEL_PLUGIN_LOG_LEVEL: "trace",
  388. };
  389. if (baseUrl) env.ANTHROPIC_BASE_URL = baseUrl;
  390. let sandbox: SandboxInstance | undefined;
  391. try {
  392. // Create sandbox from snapshot (snapshot has node runtime but global npm packages don't persist)
  393. sandbox = await Sandbox.create({ fromSnapshot: snapshotId, env, timeout: 900_000 });
  394. log(` [${project.slug}] Sandbox ${sandbox.sandboxId} ready (${elapsed(t0)})`);
  395. // Install Claude Code (global npm packages don't survive snapshot restore)
  396. const claudeInstall = await run(sandbox, "sh", ["-c", "npm install -g @anthropic-ai/claude-code"]);
  397. const claudeBin = (await run(sandbox, "sh", ["-c", "which claude"])).out;
  398. log(` [${project.slug}] Claude installed at ${claudeBin} (${elapsed(t0)})`);
  399. // Set up project directory and install plugin from local dev copy
  400. await run(sandbox, "mkdir", ["-p", projectDir]);
  401. await run(sandbox, "sh", ["-c", `cd ${projectDir} && npm init -y`]);
  402. const pluginInstall = await uploadAndInstallPlugin(sandbox, projectDir);
  403. log(` [${project.slug}] add-plugin exit=${pluginInstall.exit}`);
  404. if (pluginInstall.out) log(` [${project.slug}] add-plugin out: ${pluginInstall.out.slice(0, 200)}`);
  405. if (pluginInstall.err) log(` [${project.slug}] add-plugin err: ${pluginInstall.err.slice(0, 200)}`);
  406. if (pluginInstall.exit !== 0 && !pluginInstall.out.includes("successfully") && !pluginInstall.out.includes("Installed")) {
  407. throw new Error(`Plugin install failed (exit ${pluginInstall.exit}): ${(pluginInstall.out + " " + pluginInstall.err).slice(0, 300)}`);
  408. }
  409. // Write prompt to a temp file to avoid shell injection from prompt content
  410. const promptTmpPath = "/tmp/claude-prompt.txt";
  411. await sandbox.writeFiles([{ path: promptTmpPath, content: Buffer.from(project.prompt, "utf-8") }]);
  412. // Check for PTY wrapper
  413. const scriptCheck = await run(sandbox, "sh", ["-c", "which script 2>/dev/null && echo FOUND || echo MISSING"]);
  414. const hasScript = scriptCheck.out.includes("FOUND");
  415. // Run Claude Code session — prompt is read from temp file via cat, never interpolated
  416. const settingsPath = `${projectDir}/.claude/settings.json`;
  417. let sessionResult: { out: string; err: string; exit: number };
  418. let sessionMethod: string;
  419. if (hasScript) {
  420. sessionMethod = "script-pty";
  421. sessionResult = await run(
  422. sandbox,
  423. "sh",
  424. [
  425. "-c",
  426. `cd ${projectDir} && VERCEL_PLUGIN_LOG_LEVEL=trace CLAUDE_PLUGIN_ROOT=${SANDBOX_PLUGIN_DIR} script -qec "${SANDBOX_HOME}/.global/npm/bin/claude --dangerously-skip-permissions --settings ${settingsPath} \\"\\$(cat ${promptTmpPath})\\"" /dev/null`,
  427. ],
  428. { timeout: TIMEOUT_MS },
  429. );
  430. } else {
  431. sessionMethod = "direct";
  432. sessionResult = await run(
  433. sandbox,
  434. "sh",
  435. [
  436. "-c",
  437. `cd ${projectDir} && VERCEL_PLUGIN_LOG_LEVEL=trace CLAUDE_PLUGIN_ROOT=${SANDBOX_PLUGIN_DIR} ${SANDBOX_HOME}/.global/npm/bin/claude --dangerously-skip-permissions --settings ${settingsPath} "$(cat ${promptTmpPath})"`,
  438. ],
  439. { timeout: TIMEOUT_MS },
  440. );
  441. }
  442. const timedOut = sessionResult.exit === 124; // timeout exit code
  443. // Extract hook evidence
  444. const claimCheck = await run(sandbox, "sh", [
  445. "-c",
  446. "find /tmp -maxdepth 1 -name 'vercel-plugin-*-seen-skills.d' -type d 2>/dev/null | head -5",
  447. ]);
  448. const claimDirs = claimCheck.out.split("\n").filter(Boolean);
  449. let claimedSkills: string[] = [];
  450. if (claimDirs.length > 0) {
  451. const contents = await run(sandbox, "sh", ["-c", `ls ${claimDirs[0]} 2>/dev/null`]);
  452. claimedSkills = contents.out.split("\n").filter(Boolean);
  453. }
  454. const seenFileCheck = await run(sandbox, "sh", [
  455. "-c",
  456. "find /tmp -maxdepth 1 -name 'vercel-plugin-*-seen-skills.txt' 2>/dev/null | head -1",
  457. ]);
  458. const debugLogCheck = await run(sandbox, "sh", [
  459. "-c",
  460. `find /root/.claude/debug -name '*.txt' -o -name '*.log' 2>/dev/null; find ${SANDBOX_HOME}/.claude/debug -name '*.txt' -o -name '*.log' 2>/dev/null`,
  461. ]);
  462. const debugLogFiles = debugLogCheck.out.split("\n").filter(Boolean);
  463. const hookEvidence = {
  464. claimDirs: claimDirs.length > 0,
  465. seenFile: seenFileCheck.out.length > 0,
  466. debugLogCount: debugLogFiles.length,
  467. preToolUseInStderr:
  468. sessionResult.err.includes("PreToolUse") ||
  469. sessionResult.err.includes("pretooluse") ||
  470. sessionResult.err.includes("skill-inject"),
  471. userPromptInStderr:
  472. sessionResult.err.includes("UserPromptSubmit") ||
  473. sessionResult.err.includes("user-prompt-submit"),
  474. };
  475. // Save artifacts to results dir
  476. const slugDir = join(RESULTS_DIR, runId, project.slug);
  477. await mkdir(slugDir, { recursive: true });
  478. await mkdir(join(slugDir, "claim-dir"), { recursive: true });
  479. await mkdir(join(slugDir, "debug-logs"), { recursive: true });
  480. await writeFile(join(slugDir, "claude-output.txt"), sessionResult.out);
  481. await writeFile(join(slugDir, "stderr-trace.txt"), sessionResult.err);
  482. // Copy claim dir contents
  483. for (const skill of claimedSkills) {
  484. await writeFile(join(slugDir, "claim-dir", skill), "");
  485. }
  486. // Extract seen-skills.txt
  487. if (seenFileCheck.out.length > 0) {
  488. const seenContents = await run(sandbox, "cat", [seenFileCheck.out.split("\n")[0]]);
  489. await writeFile(join(slugDir, "seen-skills.txt"), seenContents.out);
  490. }
  491. // Extract debug logs
  492. for (const logFile of debugLogFiles.slice(0, 5)) {
  493. try {
  494. const content = await run(sandbox, "sh", ["-c", `head -500 '${logFile}'`]);
  495. const basename = logFile.split("/").pop() ?? "unknown.txt";
  496. await writeFile(join(slugDir, "debug-logs", basename), content.out);
  497. } catch { /* skip unreadable logs */ }
  498. }
  499. // Project tree
  500. const tree = await run(sandbox, "sh", [
  501. "-c",
  502. `find ${projectDir} -maxdepth 3 -not -path '*/node_modules/*' -not -path '*/.git/*' | head -80`,
  503. ]);
  504. await writeFile(join(slugDir, "project-tree.txt"), tree.out);
  505. // Run meta
  506. const result: ScenarioResult = {
  507. slug: project.slug,
  508. sandboxId: sandbox.sandboxId,
  509. success: sessionResult.exit === 0 && !timedOut,
  510. timedOut,
  511. durationMs: performance.now() - t0,
  512. sessionMethod,
  513. expectedSkills: project.expectedSkills,
  514. claimedSkills,
  515. hookEvidence,
  516. };
  517. await writeFile(join(slugDir, "run-meta.json"), JSON.stringify(result, null, 2));
  518. return result;
  519. } catch (err: any) {
  520. return {
  521. slug: project.slug,
  522. sandboxId: sandbox?.sandboxId ?? "unknown",
  523. success: false,
  524. timedOut: false,
  525. durationMs: performance.now() - t0,
  526. sessionMethod: "none",
  527. expectedSkills: project.expectedSkills,
  528. claimedSkills: [],
  529. hookEvidence: {
  530. claimDirs: false,
  531. seenFile: false,
  532. debugLogCount: 0,
  533. preToolUseInStderr: false,
  534. userPromptInStderr: false,
  535. },
  536. error: err.message?.slice(0, 400) ?? String(err),
  537. };
  538. } finally {
  539. if (sandbox) {
  540. try {
  541. await sandbox.stop();
  542. } catch { /* already stopped */ }
  543. }
  544. }
  545. }
  546. // ---------------------------------------------------------------------------
  547. // Parallel orchestrator
  548. // ---------------------------------------------------------------------------
  549. async function runParallel(
  550. scenarios: BenchmarkProject[],
  551. snapshotId: string,
  552. concurrency: number,
  553. ): Promise<ScenarioResult[]> {
  554. const apiKey = resolveApiKey();
  555. const baseUrl = resolveBaseUrl();
  556. const results: ScenarioResult[] = [];
  557. const queue = [...scenarios];
  558. async function worker(): Promise<void> {
  559. while (queue.length > 0) {
  560. const project = queue.shift()!;
  561. log(`\n--- ${project.slug} (${scenarios.length - queue.length}/${scenarios.length}) ---`);
  562. const result = await runScenario(project, snapshotId, apiKey, baseUrl);
  563. results.push(result);
  564. const status = result.timedOut ? "TIMEOUT" : result.success ? "OK" : "FAIL";
  565. log(
  566. ` [${project.slug}] ${status} | skills: ${result.claimedSkills.join(", ") || "none"} | ${(result.durationMs / 1000).toFixed(1)}s`,
  567. );
  568. }
  569. }
  570. // Launch N workers
  571. const workers = Array.from({ length: Math.min(concurrency, scenarios.length) }, () => worker());
  572. await Promise.all(workers);
  573. return results;
  574. }
  575. // ---------------------------------------------------------------------------
  576. // Main
  577. // ---------------------------------------------------------------------------
  578. // runId is declared at the top alongside the Logger; no re-declaration needed here.
  579. async function main() {
  580. const t0 = performance.now();
  581. log("=== Benchmark Sandbox Runner ===\n");
  582. logger.event("runner.start", { mode: flags.scenario ? "single" : flags.quick ? "quick" : "full" });
  583. // Step 1: Create or reuse snapshot (cached for 24h, verified before use)
  584. let snapshotId: string | null = null;
  585. let snapshotCached = false;
  586. let snapshotAgeHours: number | undefined;
  587. let snapshotCreationMs: number | undefined;
  588. if (!flags["force-snapshot"]) {
  589. snapshotId = await loadCachedSnapshot();
  590. if (snapshotId) {
  591. logger.event("snapshot.cache.hit", { snapshot_id: snapshotId });
  592. try {
  593. const raw = await readFile(SNAPSHOT_CACHE_PATH, "utf-8");
  594. const cache = JSON.parse(raw) as SnapshotCache;
  595. snapshotAgeHours = (Date.now() - new Date(cache.createdAt).getTime()) / 3600_000;
  596. } catch { /* ignore */ }
  597. // Verify cached snapshot is still usable
  598. const verified = await verifySnapshot(snapshotId);
  599. if (verified) {
  600. snapshotCached = true;
  601. } else {
  602. // Stale — invalidate and recreate once
  603. const staleErr = new RunnerError(
  604. "SNAPSHOT_CACHE_STALE",
  605. `Cached snapshot ${snapshotId} failed verification`,
  606. );
  607. warn(`[snapshot] ${staleErr.message} — ${staleErr.hint}`);
  608. logger.event("snapshot.verify.failed", { snapshot_id: snapshotId, error_code: staleErr.code });
  609. await removeCachedSnapshot();
  610. snapshotId = null; // fall through to creation
  611. }
  612. }
  613. }
  614. if (!snapshotId) {
  615. // Remove stale cache before attempting creation so a failure doesn't leave one
  616. await removeCachedSnapshot();
  617. logger.event("snapshot.cache.miss", {});
  618. logger.event("snapshot.create.started", {});
  619. const snapT0 = performance.now();
  620. snapshotId = await createSnapshot();
  621. snapshotCreationMs = performance.now() - snapT0;
  622. logger.event("snapshot.create.succeeded", { snapshot_id: snapshotId, duration_ms: Math.round(snapshotCreationMs) });
  623. // Verify newly created snapshot before caching
  624. const newVerified = await verifySnapshot(snapshotId);
  625. if (!newVerified) {
  626. const verifyErr = new RunnerError(
  627. "SNAPSHOT_VERIFY_FAILED",
  628. `Newly created snapshot ${snapshotId} failed verification`,
  629. );
  630. logger.error(`Fatal: ${verifyErr.message}`);
  631. logger.error(`Hint: ${verifyErr.hint}`);
  632. logger.event("runner.failed", { error: verifyErr.message, error_code: verifyErr.code });
  633. process.exit(EXIT_FATAL);
  634. }
  635. await saveCachedSnapshot(snapshotId);
  636. }
  637. log(`\nSnapshot: ${snapshotId}\n`);
  638. if (flags["snapshot-only"]) {
  639. log("--snapshot-only: done.");
  640. process.exit(0);
  641. }
  642. // Step 2: Select scenarios
  643. let scenarios = SCENARIOS;
  644. if (flags.scenario) {
  645. const match = SCENARIOS.find((s) => s.slug === flags.scenario);
  646. if (!match) {
  647. logger.error(`Unknown scenario: ${flags.scenario}`);
  648. logger.error(`Available: ${SCENARIOS.map((s) => s.slug).join(", ")}`);
  649. process.exit(1);
  650. }
  651. scenarios = [match];
  652. } else if (flags.quick) {
  653. scenarios = SCENARIOS.filter((s) => TIER1_SLUGS.has(s.slug));
  654. }
  655. log(`Scenarios: ${scenarios.length} (${flags.quick ? "quick" : flags.scenario ?? "full"})`);
  656. log(`Concurrency: ${CONCURRENCY}`);
  657. log(`Timeout: ${TIMEOUT_MS / 1000}s per scenario`);
  658. log(`Results: ${RESULTS_DIR}/${runId}\n`);
  659. // Create results directory
  660. await mkdir(join(RESULTS_DIR, runId), { recursive: true });
  661. // Step 3: Run scenarios
  662. const results = await runParallel(scenarios, snapshotId, CONCURRENCY);
  663. // Step 4: Build RunSummary (single source of truth for both human + JSON output)
  664. const scenarioSummaries: ScenarioSummary[] = results.map((r) => ({
  665. slug: r.slug,
  666. sandbox_id: r.sandboxId,
  667. status: scenarioStatus(r),
  668. duration_ms: r.durationMs,
  669. session_method: r.sessionMethod,
  670. expected_skills: r.expectedSkills,
  671. claimed_skills: r.claimedSkills,
  672. hook_evidence: {
  673. claim_dirs: r.hookEvidence.claimDirs,
  674. seen_file: r.hookEvidence.seenFile,
  675. debug_log_count: r.hookEvidence.debugLogCount,
  676. pre_tool_use_in_stderr: r.hookEvidence.preToolUseInStderr,
  677. user_prompt_in_stderr: r.hookEvidence.userPromptInStderr,
  678. },
  679. error: r.error,
  680. }));
  681. const snapshotMeta: SnapshotMeta = {
  682. snapshot_id: snapshotId,
  683. cached: snapshotCached,
  684. ...(snapshotAgeHours !== undefined && { age_hours: Math.round(snapshotAgeHours * 10) / 10 }),
  685. ...(snapshotCreationMs !== undefined && { creation_duration_ms: Math.round(snapshotCreationMs) }),
  686. };
  687. const mode = flags.scenario ? "single" as const : flags.quick ? "quick" as const : "full" as const;
  688. const summary: RunSummary = {
  689. schema_version: 1,
  690. run_id: runId,
  691. status: overallStatus(scenarioSummaries),
  692. timestamp: new Date().toISOString(),
  693. snapshot: snapshotMeta,
  694. scenarios: scenarioSummaries,
  695. timing: {
  696. total_duration_ms: Math.round(performance.now() - t0),
  697. scenario_durations_ms: Object.fromEntries(
  698. scenarioSummaries.map((s) => [s.slug, Math.round(s.duration_ms)]),
  699. ),
  700. },
  701. config: {
  702. concurrency: CONCURRENCY,
  703. timeout_ms: TIMEOUT_MS,
  704. results_dir: join(RESULTS_DIR, runId),
  705. mode,
  706. },
  707. };
  708. // Write manifest (always, regardless of --json)
  709. await writeFile(join(RESULTS_DIR, runId, "run-manifest.json"), JSON.stringify(summary, null, 2));
  710. // Step 5: Print human summary table (derived from RunSummary)
  711. log("\n=== Summary ===");
  712. log(
  713. `${"Slug".padEnd(28)} ${"Status".padEnd(10)} ${"Method".padEnd(12)} ${"Skills".padEnd(30)} Duration`,
  714. );
  715. log("-".repeat(100));
  716. for (const s of summary.scenarios) {
  717. const dur = `${(s.duration_ms / 1000).toFixed(1)}s`;
  718. const skills = s.claimed_skills.join(", ") || "(none)";
  719. log(`${s.slug.padEnd(28)} ${s.status.toUpperCase().padEnd(10)} ${s.session_method.padEnd(12)} ${skills.padEnd(30)} ${dur}`);
  720. }
  721. const passed = summary.scenarios.filter((s) => s.status === "pass").length;
  722. const hookFired = summary.scenarios.filter(
  723. (s) => s.hook_evidence.claim_dirs || s.hook_evidence.pre_tool_use_in_stderr,
  724. ).length;
  725. log(`\n${passed}/${summary.scenarios.length} scenarios succeeded`);
  726. log(`${hookFired}/${summary.scenarios.length} scenarios had hook evidence`);
  727. log(`Results: ${summary.config.results_dir}`);
  728. log(`Total time: ${elapsed(t0)}\n`);
  729. // --json: write only RunSummary to stdout (everything else already went to stderr)
  730. if (JSON_MODE) {
  731. process.stdout.write(JSON.stringify(summary, null, 2) + "\n");
  732. }
  733. // Exit codes: 0=all pass, 1=some fail, 2=fatal
  734. logger.event("runner.completed", { status: summary.status, scenarios: summary.scenarios.length });
  735. const exitCode = summary.status === "pass" ? EXIT_ALL_PASS : EXIT_SOME_FAIL;
  736. process.exit(exitCode);
  737. }
  738. main().catch((err) => {
  739. logger.error(`Fatal: ${err.message ?? err}`);
  740. logger.event("runner.failed", { error: err.message ?? String(err) });
  741. process.exit(EXIT_FATAL);
  742. });
.claude/skills/benchmark-sandbox/spike/interactive-proof.ts · Bundled script · 461 lines · 2 findings marked
  1. #!/usr/bin/env bun
  2. /**
  3. * Spike: Prove interactive Claude Code session with hooks firing
  4. *
  5. * Acceptance criteria:
  6. * 1. Claude Code session starts inside sandbox and processes a prompt
  7. * 2. PreToolUse hook fires (verified by debug log or claim dir presence)
  8. * 3. UserPromptSubmit hook fires (verified by debug log)
  9. * 4. Session completes without hanging or falling back to non-interactive mode
  10. * 5. Debug logs and claim dir contents are extractable via readFile()
  11. *
  12. * Strategy:
  13. * - Create sandbox with node24 + API credentials
  14. * - Install Claude Code globally
  15. * - Install vercel-plugin via npx add-plugin
  16. * - Attempt interactive session with PTY wrapper (`script` command)
  17. * - Fall back to direct runCommand if PTY wrapper unavailable
  18. * - Extract debug logs + claim dir contents for verification
  19. */
  20. import { Sandbox } from "@vercel/sandbox";
  21. // ── helpers ──────────────────────────────────────────────────────────
  22. function resolveApiKey(): string {
  23. const key =
  24. process.env.ANTHROPIC_API_KEY ??
  25. process.env.AI_GATEWAY_API_KEY ??
  26. process.env.VERCEL_API_KEY;
  27. if (!key) {
  28. console.error(
  29. "Missing API key. Set one of: ANTHROPIC_API_KEY, AI_GATEWAY_API_KEY, VERCEL_API_KEY"
  30. );
  31. process.exit(1);
  32. }
  33. return key;
  34. }
  35. function elapsed(start: number): string {
  36. return `${((performance.now() - start) / 1000).toFixed(1)}s`;
  37. }
  38. type CmdResult = Awaited<
  39. ReturnType<InstanceType<typeof Sandbox>["runCommand"]>
  40. >;
  41. async function run(
  42. sandbox: InstanceType<typeof Sandbox>,
  43. cmd: string,
  44. args: string[],
  45. opts?: { timeout?: number; env?: Record<string, string> }
  46. ): Promise<{ out: string; err: string; exit: number }> {
  47. const result: CmdResult = await sandbox.runCommand(cmd, args, opts);
  48. return {
  49. out: (await result.stdout()).trim(),
  50. err: (await result.stderr()).trim(),
  51. exit: (result as any).exitCode ?? 0,
  52. };
  53. }
  54. interface StepResult {
  55. step: string;
  56. ok: boolean;
  57. detail: string;
  58. time: string;
  59. }
  60. // ── test prompt ──────────────────────────────────────────────────────
  61. // Deliberately product-focused (not tech-focused) to exercise skill matching.
  62. // Should trigger: nextjs, ai-sdk at minimum.
  63. const TEST_PROMPT = `Create a Next.js app with a single page that uses the AI SDK to stream a response from a language model. The page should have an input field and a submit button. Keep it minimal.`;
  64. // ── main ─────────────────────────────────────────────────────────────
  65. async function main() {
  66. const t0 = performance.now();
  67. const apiKey = resolveApiKey();
  68. const baseUrl = process.env.ANTHROPIC_BASE_URL;
  69. const results: StepResult[] = [];
  70. let sandbox: InstanceType<typeof Sandbox> | undefined;
  71. try {
  72. // ── Step 1: Create sandbox ───────────────────────────────────────
  73. console.log("[1/7] Creating sandbox (node24)...");
  74. const t1 = performance.now();
  75. const sandboxEnv: Record<string, string> = {
  76. ANTHROPIC_API_KEY: apiKey,
  77. // Enable trace-level hook logging so we can verify hook firing
  78. VERCEL_PLUGIN_LOG_LEVEL: "trace",
  79. };
  80. if (baseUrl) sandboxEnv.ANTHROPIC_BASE_URL = baseUrl;
  81. sandbox = await Sandbox.create({ runtime: "node24", env: sandboxEnv });
  82. results.push({
  83. step: "create-sandbox",
  84. ok: true,
  85. detail: `id=${sandbox.sandboxId}`,
  86. time: elapsed(t1),
  87. });
  88. console.log(` OK: ${sandbox.sandboxId} (${elapsed(t1)})`);
  89. // ── Step 2: Install Claude Code ──────────────────────────────────
  90. console.log("[2/7] Installing Claude Code...");
  91. const t2 = performance.now();
  92. const install = await run(sandbox, "npm", [
  93. "install",
  94. "-g",
  95. "@anthropic-ai/claude-code",
  96. ]);
  97. results.push({
  98. step: "install-claude-code",
  99. ok: install.exit === 0,
  100. detail:
  101. install.exit === 0
  102. ? "exit=0"
  103. : `exit=${install.exit} err=${install.err.slice(0, 200)}`,
  104. time: elapsed(t2),
  105. });
  106. if (install.exit !== 0) throw new Error(`Claude Code install failed: ${install.err.slice(0, 300)}`);
  107. console.log(` OK (${elapsed(t2)})`);
  108. // ── Step 3: Scaffold project + install plugin ────────────────────
  109. console.log("[3/7] Scaffolding project and installing plugin...");
  110. const t3 = performance.now();
  111. const projectDir = "/home/user/test-project";
  112. // Create a minimal project so hooks have a project context
  113. await run(sandbox, "mkdir", ["-p", projectDir]);
  114. await run(sandbox, "sh", [
  115. "-c",
  116. `cd ${projectDir} && npm init -y && npm install next react react-dom`,
  117. ]);
  118. // Install vercel-plugin into the project
  119. const pluginInstall = await run(sandbox, "sh", [
  120. "-c",
  121. `cd ${projectDir} && npx -y add-plugin https://github.com/vercel/vercel-plugin -s project -y`,
  122. ]);
  123. const pluginOk =
  124. pluginInstall.exit === 0 ||
  125. pluginInstall.out.includes("successfully") ||
  126. pluginInstall.out.includes("added");
  127. results.push({
  128. step: "install-plugin",
  129. ok: pluginOk,
  130. detail: pluginOk
  131. ? "plugin installed"
  132. : `exit=${pluginInstall.exit} out=${pluginInstall.out.slice(0, 200)}`,
  133. time: elapsed(t3),
  134. });
  135. if (!pluginOk) throw new Error(`Plugin install failed: ${pluginInstall.out.slice(0, 300)}`);
  136. console.log(` OK (${elapsed(t3)})`);
  137. // Verify hooks.json exists
  138. const hooksCheck = await run(sandbox, "sh", [
  139. "-c",
  140. `test -f ${projectDir}/.claude/hooks.json && echo EXISTS || echo MISSING`,
  141. ]);
  142. console.log(` hooks.json: ${hooksCheck.out}`);
  143. // ── Step 4: Check for PTY wrapper availability ───────────────────
  144. console.log("[4/7] Checking PTY wrapper availability...");
  145. const t4 = performance.now();
  146. const scriptCheck = await run(sandbox, "sh", [
  147. "-c",
  148. "which script 2>/dev/null && echo FOUND || echo MISSING",
  149. ]);
  150. const hasScript = scriptCheck.out.includes("FOUND");
  151. // Also check for unbuffer as alternative
  152. const unbufferCheck = await run(sandbox, "sh", [
  153. "-c",
  154. "which unbuffer 2>/dev/null && echo FOUND || echo MISSING",
  155. ]);
  156. const hasUnbuffer = unbufferCheck.out.includes("FOUND");
  157. results.push({
  158. step: "pty-check",
  159. ok: true, // informational
  160. detail: `script=${hasScript ? "yes" : "no"} unbuffer=${hasUnbuffer ? "yes" : "no"}`,
  161. time: elapsed(t4),
  162. });
  163. console.log(
  164. ` script: ${hasScript ? "yes" : "no"}, unbuffer: ${hasUnbuffer ? "yes" : "no"}`
  165. );
  166. // ── Step 5: Run interactive Claude Code session ──────────────────
  167. console.log("[5/7] Running Claude Code session...");
  168. const t5 = performance.now();
  169. // Build the claude command. Key flags:
  170. // --dangerously-skip-permissions: avoids interactive permission prompts
  171. // --settings: point to plugin settings
  172. // The prompt is passed as a positional argument
  173. //
  174. // We try multiple approaches in order:
  175. // 1. `script` PTY wrapper (most likely to trigger hooks)
  176. // 2. Direct `claude` invocation (may work if SDK detects session context)
  177. const settingsPath = `${projectDir}/.claude/settings.json`;
  178. const escapedPrompt = TEST_PROMPT.replace(/'/g, "'\\''");
  179. let sessionResult: { out: string; err: string; exit: number };
  180. let sessionMethod: string;
  181. if (hasScript) {
  182. // `script -qec` wraps the command in a PTY on Linux
  183. // -q: quiet (no "Script started" banner)
  184. // -e: return exit code of child
  185. // -c: command to run
  186. sessionMethod = "script-pty";
  187. const scriptCmd = [
  188. "-c",
  189. `cd ${projectDir} && VERCEL_PLUGIN_LOG_LEVEL=trace script -qec 'claude --dangerously-skip-permissions --verbose "${escapedPrompt}"' /dev/null`,
  190. ];
  191. sessionResult = await run(sandbox, "sh", scriptCmd, {
  192. timeout: 120_000, // 2 min for a simple prompt
  193. });
  194. } else {
  195. // Direct invocation — hooks may or may not fire
  196. sessionMethod = "direct";
  197. sessionResult = await run(
  198. sandbox,
  199. "sh",
  200. [
  201. "-c",
  202. `cd ${projectDir} && claude --dangerously-skip-permissions --verbose '${escapedPrompt}'`,
  203. ],
  204. { timeout: 120_000 }
  205. );
  206. }
  207. const sessionOk = sessionResult.exit === 0;
  208. results.push({
  209. step: "claude-session",
  210. ok: sessionOk,
  211. detail: `method=${sessionMethod} exit=${sessionResult.exit} out_len=${sessionResult.out.length} err_len=${sessionResult.err.length}`,
  212. time: elapsed(t5),
  213. });
  214. console.log(
  215. ` ${sessionOk ? "OK" : "FAIL"}: method=${sessionMethod} exit=${sessionResult.exit} (${elapsed(t5)})`
  216. );
  217. if (sessionResult.out.length > 0) {
  218. console.log(` stdout preview: ${sessionResult.out.slice(0, 300)}`);
  219. }
  220. if (sessionResult.err.length > 0) {
  221. console.log(` stderr preview: ${sessionResult.err.slice(0, 300)}`);
  222. }
  223. // ── Step 6: Verify hook firing ───────────────────────────────────
  224. console.log("[6/7] Verifying hook firing...");
  225. const t6 = performance.now();
  226. // Check 6a: Look for claim dir (dedup state = hooks wrote it)
  227. const claimCheck = await run(sandbox, "sh", [
  228. "-c",
  229. `find /tmp -maxdepth 1 -name 'vercel-plugin-*-seen-skills.d' -type d 2>/dev/null | head -5`,
  230. ]);
  231. const claimDirs = claimCheck.out
  232. .split("\n")
  233. .filter((l) => l.length > 0);
  234. const hasClaimDirs = claimDirs.length > 0;
  235. // If claim dirs exist, list their contents (= which skills were injected)
  236. let claimedSkills: string[] = [];
  237. if (hasClaimDirs) {
  238. const claimContents = await run(sandbox, "sh", [
  239. "-c",
  240. `ls ${claimDirs[0]} 2>/dev/null`,
  241. ]);
  242. claimedSkills = claimContents.out
  243. .split("\n")
  244. .filter((l) => l.length > 0)
  245. .map((s) => decodeURIComponent(s));
  246. }
  247. // Check 6b: Look for seen-skills.txt file
  248. const seenFileCheck = await run(sandbox, "sh", [
  249. "-c",
  250. `find /tmp -maxdepth 1 -name 'vercel-plugin-*-seen-skills.txt' 2>/dev/null | head -3`,
  251. ]);
  252. const hasSeenFile = seenFileCheck.out.length > 0;
  253. let seenFileContents = "";
  254. if (hasSeenFile) {
  255. const firstFile = seenFileCheck.out.split("\n")[0];
  256. const contents = await run(sandbox, "cat", [firstFile]);
  257. seenFileContents = contents.out;
  258. }
  259. // Check 6c: Look for debug logs
  260. const debugLogCheck = await run(sandbox, "sh", [
  261. "-c",
  262. `find /root/.claude/debug -name '*.txt' -o -name '*.log' 2>/dev/null; find /home/user/.claude/debug -name '*.txt' -o -name '*.log' 2>/dev/null`,
  263. ]);
  264. const debugLogFiles = debugLogCheck.out
  265. .split("\n")
  266. .filter((l) => l.length > 0);
  267. // Check 6d: Grep stderr for hook trace markers
  268. const preToolUseInStderr = sessionResult.err.includes("PreToolUse") ||
  269. sessionResult.err.includes("pretooluse") ||
  270. sessionResult.err.includes("skill-inject");
  271. const userPromptInStderr = sessionResult.err.includes("UserPromptSubmit") ||
  272. sessionResult.err.includes("user-prompt-submit") ||
  273. sessionResult.err.includes("prompt-signal");
  274. const hookEvidence = {
  275. claimDirs: hasClaimDirs,
  276. claimedSkills,
  277. seenFile: hasSeenFile,
  278. seenFileContents,
  279. debugLogFiles,
  280. preToolUseInStderr,
  281. userPromptInStderr,
  282. };
  283. // A hook "fired" if we see any evidence at all
  284. const anyHookEvidence =
  285. hasClaimDirs ||
  286. hasSeenFile ||
  287. preToolUseInStderr ||
  288. userPromptInStderr ||
  289. debugLogFiles.length > 0;
  290. results.push({
  291. step: "hook-verification",
  292. ok: anyHookEvidence,
  293. detail: JSON.stringify(hookEvidence, null, 0),
  294. time: elapsed(t6),
  295. });
  296. console.log(
  297. ` ${anyHookEvidence ? "OK" : "FAIL"}: hook evidence found=${anyHookEvidence}`
  298. );
  299. console.log(` claim dirs: ${hasClaimDirs} (skills: ${claimedSkills.join(", ") || "none"})`);
  300. console.log(` seen-skills.txt: ${hasSeenFile} (${seenFileContents || "empty"})`);
  301. console.log(` debug logs: ${debugLogFiles.length} file(s)`);
  302. console.log(` stderr traces: PreToolUse=${preToolUseInStderr} UserPromptSubmit=${userPromptInStderr}`);
  303. // ── Step 7: Extract artifacts ────────────────────────────────────
  304. console.log("[7/7] Extracting artifacts...");
  305. const t7 = performance.now();
  306. const artifacts: Record<string, string> = {};
  307. // Extract debug log contents
  308. for (const logFile of debugLogFiles.slice(0, 3)) {
  309. try {
  310. const content = await run(sandbox, "sh", [
  311. "-c",
  312. `head -200 '${logFile}'`,
  313. ]);
  314. artifacts[logFile] = content.out;
  315. } catch {
  316. artifacts[logFile] = "(read failed)";
  317. }
  318. }
  319. // Extract project structure
  320. const tree = await run(sandbox, "sh", [
  321. "-c",
  322. `find ${projectDir} -maxdepth 3 -not -path '*/node_modules/*' -not -path '*/.git/*' | head -50`,
  323. ]);
  324. artifacts["project-tree"] = tree.out;
  325. // Extract .claude directory structure
  326. const claudeTree = await run(sandbox, "sh", [
  327. "-c",
  328. `find ${projectDir}/.claude -type f 2>/dev/null | head -20`,
  329. ]);
  330. artifacts[".claude-tree"] = claudeTree.out;
  331. // Full stderr (hook trace log)
  332. artifacts["stderr-trace"] = sessionResult.err.slice(0, 5000);
  333. // Claude output
  334. artifacts["claude-output"] = sessionResult.out.slice(0, 3000);
  335. results.push({
  336. step: "artifact-extraction",
  337. ok: true,
  338. detail: `${Object.keys(artifacts).length} artifacts extracted`,
  339. time: elapsed(t7),
  340. });
  341. console.log(` OK: ${Object.keys(artifacts).length} artifacts (${elapsed(t7)})`);
  342. // Print key artifacts
  343. console.log("\n--- Artifacts ---");
  344. console.log("\n[.claude tree]");
  345. console.log(artifacts[".claude-tree"] || "(empty)");
  346. console.log("\n[project tree]");
  347. console.log(artifacts["project-tree"]?.slice(0, 500) || "(empty)");
  348. if (artifacts["stderr-trace"]) {
  349. console.log("\n[stderr trace (first 1000 chars)]");
  350. console.log(artifacts["stderr-trace"].slice(0, 1000));
  351. }
  352. for (const [path, content] of Object.entries(artifacts)) {
  353. if (path.includes("debug") && content !== "(read failed)") {
  354. console.log(`\n[debug log: ${path} (first 500 chars)]`);
  355. console.log(content.slice(0, 500));
  356. }
  357. }
  358. } catch (err: any) {
  359. results.push({
  360. step: "fatal",
  361. ok: false,
  362. detail: err.message?.slice(0, 400) ?? String(err),
  363. time: elapsed(t0),
  364. });
  365. console.error(`Fatal error: ${err.message}`);
  366. } finally {
  367. if (sandbox) {
  368. try {
  369. console.log("\nStopping sandbox...");
  370. await sandbox.stop();
  371. console.log("Sandbox stopped");
  372. } catch {
  373. console.log("Sandbox stop failed (may already be stopped)");
  374. }
  375. }
  376. }
  377. // ── summary ────────────────────────────────────────────────────────
  378. const allPassed = results.every((r) => r.ok);
  379. const hookStep = results.find((r) => r.step === "hook-verification");
  380. const sessionStep = results.find((r) => r.step === "claude-session");
  381. console.log("\n" + "=".repeat(60));
  382. console.log(allPassed ? "ALL CHECKS PASSED" : "SOME CHECKS FAILED");
  383. console.log("=".repeat(60));
  384. for (const r of results) {
  385. const icon = r.ok ? "PASS" : "FAIL";
  386. console.log(` [${icon}] ${r.step}: ${r.detail.slice(0, 120)} (${r.time})`);
  387. }
  388. console.log(`\nTotal time: ${elapsed(t0)}`);
  389. // Verdict on the critical question
  390. console.log("\n--- VERDICT ---");
  391. if (sessionStep?.ok && hookStep?.ok) {
  392. console.log(
  393. "Interactive Claude Code session WORKS in sandbox with hooks firing."
  394. );
  395. console.log("Proceed to full sister skill implementation.");
  396. } else if (sessionStep?.ok && !hookStep?.ok) {
  397. console.log(
  398. "Claude Code session completed but NO hook evidence found."
  399. );
  400. console.log(
  401. "Hooks may not fire in sandbox. Investigate PTY requirements or alternative session modes."
  402. );
  403. } else {
  404. console.log(
  405. "Claude Code session did NOT complete successfully."
  406. );
  407. console.log(
  408. "Review error output above. May need different install approach or sandbox configuration."
  409. );
  410. }
  411. process.exit(allPassed ? 0 : 1);
  412. }
  413. main();

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.