skill-optimizer
Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/fastxyz--skill-optimizer)HTML
<a href="/listings/fastxyz--skill-optimizer"><img src="/api/listings/fastxyz--skill-optimizer/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
Installation differs by agent. The canonical skill is `skills/skill-optimizer/SKILL.md`; every plugin manifest points at that same file.
Claude Code
Register this repository as a Claude Code plugin marketplace:
/plugin marketplace add fastxyz/skill-optimizer
Then install the plugin:
/plugin install skill-optimizer@skill-optimizer
OpenAI Codex CLI
Register this repository as a Codex plugin marketplace:
codex plugin marketplace add fastxyz/skill-optimizer
Then open the plugin search interface:
/plugins
Select `skill-optimizer` and install it.
OpenAI Codex App
In the Codex app, open Plugins from the sidebar, search for `skill-optimizer`, and install it from the Coding section.
If it is not listed, install it from Codex CLI first:
codex plugin marketplace add fastxyz/skill-optimizer
Cursor
Install the skill with the open skills CLI:
npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a cursor -y
Cursor can also import the skill from GitHub via Settings -> Rules -> Project Rules -> Add Rule -> Remote Rule (Github). The Cursor plugin metadata lives at `.cursor-plugin/plugin.json`.
OpenCode
Tell OpenCode:
Fetch and follow instructions from https://raw.githubusercontent.com/fastxyz/skill-optimizer/refs/heads/main/.opencode/INSTALL.md
Or add the plugin to `opencode.json` at user or project scope:
{
"plugin": ["skill-optimizer@git+https://github.com/fastxyz/skill-optimizer.git"]
}Restart OpenCode. See `docs/README.opencode.md` for details.
Gemini CLI
Install the Gemini extension from GitHub:
gemini extensions install https://github.com/fastxyz/skill-optimizer
To update:
gemini extensions update skill-optimizer
Skill-Only Install
If you only want the skill files without plugin metadata, use the open skills CLI:
npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a claude-code -a opencode -a codex -a cursor -y
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json · Manifest · 21 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- {
- "name": "skill-optimizer",
- "description": "Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs",
- "owner": {
- "name": "Fast"
- },
- "plugins": [
- {
- "name": "skill-optimizer",
- "description": "Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs",
- "version": "2.0.0",
- "source": "./",
- "author": {
- "name": "Fast"
- },
- "skills": [
- "./skills/skill-optimizer"
- ]
- }
- ]
- }
README.md · README · 182 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- # skill-optimizer
- Docker workbench and Agent Skill for running deterministic evals against agent skills.
- Use this repo in two ways:
- - Install the `skill-optimizer` skill/plugin into your agent so it can author and debug eval suites.
- - Run the local CLI to execute cases and suites in Docker against OpenRouter models.
- ## Installation
- Installation differs by agent. The canonical skill is `skills/skill-optimizer/SKILL.md`; every plugin manifest points at that same file.
- ### Claude Code
- Register this repository as a Claude Code plugin marketplace:
- ```text
- /plugin marketplace add fastxyz/skill-optimizer
- ```
- Then install the plugin:
- ```text
- /plugin install skill-optimizer@skill-optimizer
- ```
- ### OpenAI Codex CLI
- Register this repository as a Codex plugin marketplace:
- ```bash
- codex plugin marketplace add fastxyz/skill-optimizer
- ```
- Then open the plugin search interface:
- ```text
- /plugins
- ```
- Select `skill-optimizer` and install it.
- ### OpenAI Codex App
- In the Codex app, open Plugins from the sidebar, search for `skill-optimizer`, and install it from the Coding section.
- If it is not listed, install it from Codex CLI first:
- ```bash
- codex plugin marketplace add fastxyz/skill-optimizer
- ```
- ### Cursor
- Install the skill with the open skills CLI:
- ```bash
- npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a cursor -y
- ```
- Cursor can also import the skill from GitHub via Settings -> Rules -> Project Rules -> Add Rule -> Remote Rule (Github). The Cursor plugin metadata lives at `.cursor-plugin/plugin.json`.
- ### OpenCode
- Tell OpenCode:
- ```text
- Fetch and follow instructions from https://raw.githubusercontent.com/fastxyz/skill-optimizer/refs/heads/main/.opencode/INSTALL.md
- ```
- Or add the plugin to `opencode.json` at user or project scope:
- ```json
- {
- "plugin": ["skill-optimizer@git+https://github.com/fastxyz/skill-optimizer.git"]
- }
- ```
- Restart OpenCode. See `docs/README.opencode.md` for details.
- ### Gemini CLI
- Install the Gemini extension from GitHub:
- ```bash
- gemini extensions install https://github.com/fastxyz/skill-optimizer
- ```
- To update:
- ```bash
- gemini extensions update skill-optimizer
- ```
- ### Skill-Only Install
- If you only want the skill files without plugin metadata, use the open skills CLI:
- ```bash
- npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a claude-code -a opencode -a codex -a cursor -y
- ```
- ## Local CLI Setup
- Requirements:
- - Node.js 20+
- - Docker
- - `OPENROUTER_API_KEY` for real model runs
- Install and build:
- ```bash
- npm install
- npm run build
- ```
- Only `openrouter/...` model refs are supported.
- ## Quick Start
- Run the suite against the models listed in `suite.yml`:
- ```bash
- npx tsx src/cli.ts run-suite examples/workbench/pdf/suite.yml --trials 1
- ```
- Run one case directly:
- ```bash
- npx tsx src/cli.ts run-case ./case.yml --model openrouter/google/gemini-2.5-flash
- ```
- CLI help:
- ```bash
- npx tsx src/cli.ts --help
- npx tsx src/cli.ts run-case --help
- npx tsx src/cli.ts run-suite --help
- ```
- ## How The Workbench Works
- The workbench gives an agent a skill/reference folder, an isolated `/work` directory, and deterministic graders. It is designed for evals where success can be verified from files, command logs, SQL, generated artifacts, or other local state.
- Core concepts:
- - A case is one user-like task plus one or more graders.
- - A suite is a matrix of cases and OpenRouter models.
- - `references/` is copied into `/work`; this is where the skill under test lives.
- - The agent phase sees only `/work`, not graders, hidden answers, `/case`, or `/results`.
- - Graders run after the agent with `$CASE`, `$WORK`, and `$RESULTS` available.
- - Graders are the acceptance contract. They can inspect workspace files and artifacts, `answer.json`, `trace.jsonl`, and result state under `$RESULTS`.
- Read `docs/workbench.md` for the full model: directory layout, Docker phases, graders, outputs, and debugging.
- ## Examples
- Tracked examples live under `examples/workbench/`. The PDF example includes positive PDF extraction/splitting/creation cases and a negative case that checks the agent did not read the PDF skill file for a non-PDF task. The MCP example shows a local calculator server started as a hidden Docker service and exposed through the workbench `mcp` command.
- ```bash
- npx tsx src/cli.ts run-suite examples/workbench/pdf/suite.yml --trials 1
- npx tsx src/cli.ts run-suite examples/workbench/mcp/suite.yml --trials 1
- ```
- ## Development
- ```bash
- npm run typecheck
- npm test
- npm run build
- npx tsx src/cli.ts --help
- ```
- For Docker runner or image changes:
- ```bash
- docker build -t skill-optimizer-workbench:local -f docker/workbench-runner.Dockerfile .
- ```
- Do not commit `.skill-eval/`, `.results/`, `.env`, or credentials.
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.