Skip to content
Back to search

skill-optimizer

Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/fastxyz--skill-optimizer/badge.svg)](/listings/fastxyz--skill-optimizer)

HTML

<a href="/listings/fastxyz--skill-optimizer"><img src="/api/listings/fastxyz--skill-optimizer/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Installation differs by agent. The canonical skill is `skills/skill-optimizer/SKILL.md`; every plugin manifest points at that same file.

Claude Code

Register this repository as a Claude Code plugin marketplace:

/plugin marketplace add fastxyz/skill-optimizer

Then install the plugin:

/plugin install skill-optimizer@skill-optimizer

OpenAI Codex CLI

Register this repository as a Codex plugin marketplace:

codex plugin marketplace add fastxyz/skill-optimizer

Then open the plugin search interface:

/plugins

Select `skill-optimizer` and install it.

OpenAI Codex App

In the Codex app, open Plugins from the sidebar, search for `skill-optimizer`, and install it from the Coding section.

If it is not listed, install it from Codex CLI first:

codex plugin marketplace add fastxyz/skill-optimizer

Cursor

Install the skill with the open skills CLI:

npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a cursor -y

Cursor can also import the skill from GitHub via Settings -> Rules -> Project Rules -> Add Rule -> Remote Rule (Github). The Cursor plugin metadata lives at `.cursor-plugin/plugin.json`.

OpenCode

Tell OpenCode:

Fetch and follow instructions from https://raw.githubusercontent.com/fastxyz/skill-optimizer/refs/heads/main/.opencode/INSTALL.md

Or add the plugin to `opencode.json` at user or project scope:

{
  "plugin": ["skill-optimizer@git+https://github.com/fastxyz/skill-optimizer.git"]
}

Restart OpenCode. See `docs/README.opencode.md` for details.

Gemini CLI

Install the Gemini extension from GitHub:

gemini extensions install https://github.com/fastxyz/skill-optimizer

To update:

gemini extensions update skill-optimizer

Skill-Only Install

If you only want the skill files without plugin metadata, use the open skills CLI:

npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a claude-code -a opencode -a codex -a cursor -y

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 21 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. {
  2. "name": "skill-optimizer",
  3. "description": "Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs",
  4. "owner": {
  5. "name": "Fast"
  6. },
  7. "plugins": [
  8. {
  9. "name": "skill-optimizer",
  10. "description": "Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs",
  11. "version": "2.0.0",
  12. "source": "./",
  13. "author": {
  14. "name": "Fast"
  15. },
  16. "skills": [
  17. "./skills/skill-optimizer"
  18. ]
  19. }
  20. ]
  21. }
README.md · README · 182 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # skill-optimizer
  2. Docker workbench and Agent Skill for running deterministic evals against agent skills.
  3. Use this repo in two ways:
  4. - Install the `skill-optimizer` skill/plugin into your agent so it can author and debug eval suites.
  5. - Run the local CLI to execute cases and suites in Docker against OpenRouter models.
  6. ## Installation
  7. Installation differs by agent. The canonical skill is `skills/skill-optimizer/SKILL.md`; every plugin manifest points at that same file.
  8. ### Claude Code
  9. Register this repository as a Claude Code plugin marketplace:
  10. ```text
  11. /plugin marketplace add fastxyz/skill-optimizer
  12. ```
  13. Then install the plugin:
  14. ```text
  15. /plugin install skill-optimizer@skill-optimizer
  16. ```
  17. ### OpenAI Codex CLI
  18. Register this repository as a Codex plugin marketplace:
  19. ```bash
  20. codex plugin marketplace add fastxyz/skill-optimizer
  21. ```
  22. Then open the plugin search interface:
  23. ```text
  24. /plugins
  25. ```
  26. Select `skill-optimizer` and install it.
  27. ### OpenAI Codex App
  28. In the Codex app, open Plugins from the sidebar, search for `skill-optimizer`, and install it from the Coding section.
  29. If it is not listed, install it from Codex CLI first:
  30. ```bash
  31. codex plugin marketplace add fastxyz/skill-optimizer
  32. ```
  33. ### Cursor
  34. Install the skill with the open skills CLI:
  35. ```bash
  36. npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a cursor -y
  37. ```
  38. Cursor can also import the skill from GitHub via Settings -> Rules -> Project Rules -> Add Rule -> Remote Rule (Github). The Cursor plugin metadata lives at `.cursor-plugin/plugin.json`.
  39. ### OpenCode
  40. Tell OpenCode:
  41. ```text
  42. Fetch and follow instructions from https://raw.githubusercontent.com/fastxyz/skill-optimizer/refs/heads/main/.opencode/INSTALL.md
  43. ```
  44. Or add the plugin to `opencode.json` at user or project scope:
  45. ```json
  46. {
  47. "plugin": ["skill-optimizer@git+https://github.com/fastxyz/skill-optimizer.git"]
  48. }
  49. ```
  50. Restart OpenCode. See `docs/README.opencode.md` for details.
  51. ### Gemini CLI
  52. Install the Gemini extension from GitHub:
  53. ```bash
  54. gemini extensions install https://github.com/fastxyz/skill-optimizer
  55. ```
  56. To update:
  57. ```bash
  58. gemini extensions update skill-optimizer
  59. ```
  60. ### Skill-Only Install
  61. If you only want the skill files without plugin metadata, use the open skills CLI:
  62. ```bash
  63. npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a claude-code -a opencode -a codex -a cursor -y
  64. ```
  65. ## Local CLI Setup
  66. Requirements:
  67. - Node.js 20+
  68. - Docker
  69. - `OPENROUTER_API_KEY` for real model runs
  70. Install and build:
  71. ```bash
  72. npm install
  73. npm run build
  74. ```
  75. Only `openrouter/...` model refs are supported.
  76. ## Quick Start
  77. Run the suite against the models listed in `suite.yml`:
  78. ```bash
  79. npx tsx src/cli.ts run-suite examples/workbench/pdf/suite.yml --trials 1
  80. ```
  81. Run one case directly:
  82. ```bash
  83. npx tsx src/cli.ts run-case ./case.yml --model openrouter/google/gemini-2.5-flash
  84. ```
  85. CLI help:
  86. ```bash
  87. npx tsx src/cli.ts --help
  88. npx tsx src/cli.ts run-case --help
  89. npx tsx src/cli.ts run-suite --help
  90. ```
  91. ## How The Workbench Works
  92. The workbench gives an agent a skill/reference folder, an isolated `/work` directory, and deterministic graders. It is designed for evals where success can be verified from files, command logs, SQL, generated artifacts, or other local state.
  93. Core concepts:
  94. - A case is one user-like task plus one or more graders.
  95. - A suite is a matrix of cases and OpenRouter models.
  96. - `references/` is copied into `/work`; this is where the skill under test lives.
  97. - The agent phase sees only `/work`, not graders, hidden answers, `/case`, or `/results`.
  98. - Graders run after the agent with `$CASE`, `$WORK`, and `$RESULTS` available.
  99. - Graders are the acceptance contract. They can inspect workspace files and artifacts, `answer.json`, `trace.jsonl`, and result state under `$RESULTS`.
  100. Read `docs/workbench.md` for the full model: directory layout, Docker phases, graders, outputs, and debugging.
  101. ## Examples
  102. Tracked examples live under `examples/workbench/`. The PDF example includes positive PDF extraction/splitting/creation cases and a negative case that checks the agent did not read the PDF skill file for a non-PDF task. The MCP example shows a local calculator server started as a hidden Docker service and exposed through the workbench `mcp` command.
  103. ```bash
  104. npx tsx src/cli.ts run-suite examples/workbench/pdf/suite.yml --trials 1
  105. npx tsx src/cli.ts run-suite examples/workbench/mcp/suite.yml --trials 1
  106. ```
  107. ## Development
  108. ```bash
  109. npm run typecheck
  110. npm test
  111. npm run build
  112. npx tsx src/cli.ts --help
  113. ```
  114. For Docker runner or image changes:
  115. ```bash
  116. docker build -t skill-optimizer-workbench:local -f docker/workbench-runner.Dockerfile .
  117. ```
  118. Do not commit `.skill-eval/`, `.results/`, `.env`, or credentials.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.