ako4all
Drive an agentic loop that iteratively optimizes a GPU kernel for maximum speedup. Use this skill whenever the user wants to optimize / speed up / benchmark a GPU kernel (CUDA, Triton, TileLang, C++, Python), mentions AKO / AKO4ALL / AKO4X / agentic kernel optimization, asks to "make this kernel faster", or has a kernel they want measured against a PyTorch reference. The skill handles setup, profiling (ncu), correctness checking, iteration logging, and git commits. Bootstraps a workspace in any directory the user points at.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/tongminglaic--ako4all)HTML
<a href="/listings/tongminglaic--ako4all"><img src="/api/listings/tongminglaic--ako4all/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
Clone the repo directly into Claude Code's skills directory:
git clone https://github.com/TongmingLAIC/AKO4ALL.git ~/.claude/skills/ako4all
Or, if you already have it cloned somewhere else, symlink it:
ln -s /path/to/AKO4ALL ~/.claude/skills/ako4all
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 175 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- ---
- name: ako4all
- description: Drive an agentic loop that iteratively optimizes a GPU kernel for maximum speedup. Use this skill whenever the user wants to optimize / speed up / benchmark a GPU kernel (CUDA, Triton, TileLang, C++, Python), mentions AKO / AKO4ALL / AKO4X / agentic kernel optimization, asks to "make this kernel faster", or has a kernel they want measured against a PyTorch reference. The skill handles setup, profiling (ncu), correctness checking, iteration logging, and git commits. Bootstraps a workspace in any directory the user points at.
- ---
- # AKO4ALL — Agentic Kernel Optimization
- Drive a profile → modify → benchmark → log → commit loop on a GPU kernel until it runs faster than the reference. The user provides at minimum a kernel; everything else (reference, inputs, bench script, hints) is optional.
- ## When this skill applies
- - "optimize this kernel" / "speed up this CUDA / Triton / TileLang kernel"
- - "run AKO / AKO4ALL on ..."
- - "benchmark this kernel against PyTorch"
- - "iterate on this kernel until it's faster"
- - mentions of `ncu`, kernel profiling, GPU speedup target
- Does NOT apply when:
- - User wants to *write* a new kernel from scratch with no optimization target — just write code, no loop.
- - User wants Codex / GPT to review or implement — use `codex:rescue` instead.
- - User wants generic performance advice for code that isn't a GPU kernel.
- ## First action
- Before doing anything else, establish the **workspace** — the directory the loop runs in. It is typically the user's CWD, or a subdirectory / path they name in the prompt.
- ### Inventory the workspace + prompt
- Browse the workspace (don't run a fixed checklist — look around) and read the user's prompt to identify what the loop needs:
- - **Kernel** (required) — the code to optimize
- - **Reference** (optional) — correctness golden
- - **Input data** (optional) — data files the kernel consumes (`.npz`, `.bin`, shape lists, custom formats, etc.)
- - **Knowledge** (optional) — reference materials the user wants you to consult: algorithm notes, papers, design docs, prior PRs. Typically under `knowledge/` but anywhere the user points at.
- - **Bench mode** — user-provided bench script vs. default `bench/kernelbench/` evaluator
- - **Scaffold presence** — whether `bench-wrapper.sh`, `HINTS.md`, `ITERATIONS.md`, `bench/kernelbench/` are already at workspace root
- Whether the workspace follows AKO4ALL's `source/` / `knowledge/` / `bench/` naming or some entirely different layout is **not** the signal. What matters is whether you can identify each item above with confidence.
- ### Ask only when genuinely uncertain
- If the user's prompt + filesystem give you confidence about every required item, **don't ask** — skip straight to presenting the plan. Ask only when a piece's role is genuinely ambiguous (a kernel-shaped file with no obvious reference, two files that could both be the kernel, an input data file in an unfamiliar format you need permission to wire up a custom way, etc.). When in doubt, asking is cheaper than guessing wrong.
- ### Always present the resolved plan before running anything
- Whether you asked the user or not, list back what you decided — so the user can correct you even when you didn't think you needed to ask.
- Use the format below. Bold field labels + inline-code path values + the leading emoji marker make the plan visually scannable in any terminal theme (don't flatten to a wall of prose):
- **📋 Resolved Plan**
- - **Workspace** — `<path>`
- - **Kernel** — `<path>`
- - **Reference** — `<path>` *(or none — will use original kernel)*
- - **Input data** — `<path>` *(or inline in ref, or none)*
- - **Knowledge** — `<path>` *(or none)*
- - **Bench mode** — default (KernelBench) *(or custom: `<path>`)*
- - **Scaffold to copy** — `<list of missing files>` *(or none — already present)*
- If anything still feels uncertain at this point, **stop and ask**. Otherwise proceed to Workflow.
- ### Bringing in scaffold
- When copying scaffold (`bench-wrapper.sh`, `bench/kernelbench/`, starter `HINTS.md` / `ITERATIONS.md`, `workspace.gitignore` → as `.gitignore` in the workspace) from this skill's own directory into the workspace, **do not overwrite** files that already exist — the user may have edited `HINTS.md`, or `ITERATIONS.md` may carry prior iteration history. Copy only what's missing.
- ### Persisting user-supplied hints
- The user may supply behavior directives in two ways:
- - **Inline in the prompt** — e.g., "do not use shared memory" or "prefer Triton".
- - **External file reference** — e.g., "follow rules in /tmp/x.md" or "see hints.md".
- In **both cases**, merge those directives into `HINTS.md`. It's the persistence
- layer — directives that only live in the current session's plan are lost on resume.
- ### Surfacing HINTS.md changes
- Whenever you merge directives into `HINTS.md`, tell the user explicitly what
- happened. Example phrasings:
- > "I added your 'avoid shared memory' directive from the prompt to HINTS.md."
- > "I added the 3 rules from /tmp/user-hints.md to HINTS.md."
- Without this acknowledgment the user can't tell from your reply whether you
- added, replaced, or silently dropped their directives. Always name the **source**
- ("from your prompt" / "from /tmp/x.md").
- ## Workflow
- 1. **Analyze inputs.** Building on the inventory above, confirm `class Model` and `get_inputs()` can be assembled for default bench mode; if not, **stop and ask the user**. See `bench/kernelbench/GUIDE.md` for the input assembly contract (KernelBench-format input / raw kernel / kernel + separate data file / external path patterns).
- 2. **Create branch.** `git checkout -b opt/<kernel-name>`. If the workspace isn't a git repo, init one first.
- 3. **Initialize solution.** Create `solution/` and `scripts/`. Copy the kernel implementation files into `solution/` (only the kernel itself — reference / inputs helper files stay at their resolved locations). **Do not copy or `mkdir` canonical directories** (`source/`, `input/`, etc.) when the user's files already exist elsewhere. Point bench.sh's `--ref` and `--inputs` flags at the resolved paths in place. `solution/` is the only directory the loop owns.
- 4. **Generate bench.sh.** Build the bench command with adjusted paths, pipe through `2>&1 | tee _bench_output.txt`. Replace `{{BENCH_COMMAND}}` in `bench-wrapper.sh` to produce `scripts/bench.sh`. For default bench mode the command is `python bench/kernelbench/bench.py --ref <ref> --solution solution/<kernel> [--inputs <inputs-file>] --verbose` — include `--inputs` only when inputs are defined outside the ref file. **Do not hardcode `--backend`** in the rendered command; bench.py auto-detects backend from solution source. Add `--backend` only to override the sniff (explicit HIP labelling or mixed-backend solutions). `scripts/bench.sh` is a starting template — when the bench env needs setup (conda activate, sub-env python paths, multi-CUDA toolkit selection), edit it freely; preserve only the trajectory section (LABEL/TIMESTAMP handling and `cp -r solution/* "$TRAJ_DIR/"`).
- **Common env friction:** base shell often has no `python` on PATH when python lives in a sub-env (e.g. `~/anaconda3/envs/py312/bin/python`). Tools that internally subprocess `python` (sol-execbench CLI, torch `cpp_extension.load_inline`) will then fail with `command not found`. Workaround: put `PATH=<env-bin>:$PATH` at the top of `scripts/bench.sh` (or `source <conda>/etc/profile.d/conda.sh && conda activate <env>`). Discover sub-envs via `ls /home/*/anaconda3/envs/*/bin /root/*/envs/*/bin /opt/conda/envs/*/bin 2>/dev/null`.
- 5. **Verify baseline.** Run `bash scripts/bench.sh`. Expect `CORRECT=True`. If not, diagnose and fix before iterating. Commit: `git add -A && git commit -m "[baseline] Initialize solution and benchmark"`. Then run `ncu` once on the baseline to inform iter-1 direction.
- ## Iteration protocol
- Every modification to `solution/` followed by a bench run = one iteration. Number sequentially (1, 2, 3, …). Each iter is exactly three steps:
- 1. `bash scripts/bench.sh iter-N` — label is required, must match `iter-N` format.
- 2. Append a structured entry to `ITERATIONS.md` (template inside that file).
- 3. `git commit -m "[iter N] <short description of optimization direction>"`.
- **Steps 2 and 3 MUST be the next two tool calls after step 1 — no ncu, no probes, no reads, no planning the next iter between them.** A failed or partial bench is still an iter; log + commit first, debug after. This is the most-missed step in practice: agents read the bench result and telescope into next-iter analysis (probes, ncu, hypothesis forming) without closing out the current one, leaving commit gaps with `ITERATIONS.md` entries written from memory later.
- Backstop: if you catch yourself starting a new iter (Editing `solution/`, or running ncu/probes for the next direction) and `git log -1` doesn't show `[iter N] ...`, stop and finish the prior iter's steps 2 and 3 first. Related experiments that belong together narratively get grouped in `ITERATIONS.md` analysis prose, not in batched git commits.
- Profile to identify bottlenecks — see "ncu profiling" below for the ncu workflow and analytical fallback. Do not optimize blindly.
- ## Keeping the iteration loop fast
- A bench run must be cheap enough to iterate against (seconds to low minutes). When it isn't, the cause is almost always an **expensive reference** — it's re-run for every correctness trial *and* re-timed for the speedup denominator, yet it's **invariant across solution edits**, so most of that cost is wasted. This is an eval-*time* problem, not a metric problem: never change *what* you compare against to make the bench cheaper.
- Separate the per-iteration **signal** from the final **verdict**:
- - **Signal** (every iter): rank candidates by the **solution's own `RUNTIME`** (lower is better) — the reference contributes nothing to comparing two solutions.
- - **Verdict** (before committing a winner / a `final`): a full run — full trial counts, reference measured, real `SPEEDUP`, reward-hack check.
- **Whose eval is it determines what you may touch:**
- - **Default bench** (`bench/kernelbench/bench.py` — the skill owns it): pull levers freely, cheapest-and-safest first — `--no-ref` (skip reference timing; `REF_RUNTIME`/`SPEEDUP` → -1, `COMPILED`/`CORRECT`/`RUNTIME` unaffected) → trim `--num-perf-trials` (e.g. 50→20; latency noise only) → trim `--num-correct-trials` (higher risk: weakens the fresh-input anti-cheat and still runs the ref once per trial, so keep ≥1 in the loop, full at the gate). Keep the input regime **fixed across the whole run**: `--fresh-inputs` (the default) and `--no-fresh-inputs` measure different quantities, so switching mid-run makes iterations incomparable — and the final verdict must use the same regime the loop ranked under.
- - **User-provided eval** (custom `{{BENCH_COMMAND}}`): the trial counts, correctness rounds, and reference handling are the **user's contract** — do **not** inject `--no-ref` or cut counts on a script you didn't author (it may have no such flag, break the interface, or silently invalidate the measurement / a leaderboard's required N). Use only the fast-iteration switches the user exposed (flags / env vars documented in the prompt or `HINTS.md`). If iteration is too slow and none exist, **raise it with the user** — don't fabricate one.
- Caching the reference's *runtime* across iterations is sound only on a clock-locked GPU; on unlocked clocks (ref and solution timed in different clock states) prefer ranking by the solution's own latency.
- ## Stall handling
- When 3 consecutive iterations show no improvement (≥3% over current best), pause the loop and re-assess before iter N+1. Re-assessment combines:
- - **Re-profile** with `ncu` if available, or re-read runtime stats from `ITERATIONS.md` (median vs min/mean, distribution shape) if not.
- - **WebSearch** for op-specific best-known techniques / numbers on the same hardware class.
- - **Review `ITERATIONS.md`** for patterns (which axes have been tried, which haven't, where prior wins came from).
- Default outcome: pick a new direction and continue. Only escalate to stop (see next section) if re-assessment produces concrete evidence the current state is at a physical floor.
- ## When to stop
- Legitimate triggers:
- 1. User-specified iteration cap reached (in prompt or `HINTS.md`).
- 2. Stall re-assessment produced hard evidence of a floor — e.g., min runtime at cuda-event timer resolution, kernel arithmetic at HBM bandwidth limit, launch overhead dominating compute. Cite the evidence in `ITERATIONS.md`.
- 3. All viable directions exhausted: document at least 3 distinct directions tried (with their iteration numbers) in `ITERATIONS.md` before invoking this trigger, to prevent premature stops.
- Do not stop silently because tooling is unavailable — that's a re-assessment input, not a stop reason.
- ### HEAD handling on stop
- After deciding to stop, leave HEAD at the best-performing iter — not necessarily the latest. Procedure:
- 1. Identify the best iter by reading `ITERATIONS.md` Summary, the bench output for each iter under `trajectory/`, and your own reasoning notes. Useful signals from KernelBench output: median speedup, runtime std (consistency), min runtime (tail), `CORRECT` flag. Other bench harnesses expose different shapes — use what's available. Justify your pick in the commit message (e.g., "iter 4: best mean AND lowest min, while iter 6 ties on mean but has higher std").
- 2. If best iter ≠ latest iter:
- - `git checkout <best-iter-sha> -- solution/` — verbatim copy, do NOT hand-reconstruct from memory or earlier notes.
- - `bash scripts/bench.sh final` to sanity-verify on a fresh run.
- - `git commit -m "[final] Restore iter-K (X.XXx) — <one-sentence why>"`.
- The `git checkout` step is mandatory. Manual reconstruction risks introducing silent drift from the actually-benched code.
- ## ncu profiling — best effort, not a gate
- Probe `ncu` once after baseline. If it fails (driver mismatch, missing toolkit, user opt-out via free-text `HINTS.md` directive), proceed analytically for the rest of the loop without re-probing within this session. Don't gate iteration progress on ncu availability; analytical reasoning + runtime stats from the bench harness are a valid substitute for the optimizer's direction picking.
- ## Gotchas
- - **Pursue genuine latency reduction, not reward hacking.** No CUDA stream injection to evade timing, no monkey-patching the benchmark, no returning uninitialized results. The built-in evaluator flags >10× speedups for a reason — investigate before celebrating. It also re-randomizes the timed inputs in place after timing (mutation sentinel): a solution that keys on input identity and replays a stored output fails outright.
- - **The solution file must not contain `get_inputs` / `get_init_inputs`.** The bench script strips the solution's module-level tail before eval as an anti-cheat boundary. Inputs come from the reference or `--inputs` file, never the solution.
- - **`get_inputs()` must produce fresh data each call.** Bench calls it many times — every correctness trial, and (under the default fresh regime) before every timed trial. Module-level cached tensors make correctness checks trivially pass and let timing measure cache-warm performance. Use `torch.randn` or reload from disk on every call.
- - **Don't be lazy.** Stay-in-PyTorch, only-tune-configurations, skip-profiling — these are the default low-effort failure modes for agents. The point of the loop is to *rewrite* the implementation — switch languages (Triton → CUDA, etc.) when it helps.
- ## Reference files
- - `bench/kernelbench/GUIDE.md` — full input assembly patterns, CLI flags, timing methods, tolerances. Read this before writing the bench command if anything about input shape, precision, or backend is non-obvious.
- - `HINTS.md` (workspace) — user-editable behavior directives. Read at session start; respect any constraint named there.
- - `ITERATIONS.md` (workspace) — your own iteration log. Write to it every iteration.
README.md · README · 291 lines · 1 finding marked
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- <h1 align="center">AKO4ALL</h1>
- <p align="center"><b>Agentic Kernel Optimization for All</b></p>
- <p align="center">
- <a href="https://tongminglaic.github.io/AKO"><img src="https://img.shields.io/badge/Project-Page-blue" alt="Project Page"></a>
- <a href="https://github.com/TongmingLAIC/AKO4X"><img src="https://img.shields.io/badge/GitHub-AKO4X-blue?logo=github" alt="AKO4X"></a>
- <a href="https://tongminglaic.github.io/AKO/assets/ako-tech-report.pdf"><img src="https://img.shields.io/badge/Tech%20Report-PDF-blue" alt="Tech Report"></a>
- </p>
- <p align="center"><b>If you find our work useful, please consider giving us a star 🌟</b></p>
- <p align="center">
- <img src="assets/speedup_vs_expert.png" alt="Bar chart: AKO4ALL geomean speedup over the FlashInfer expert on four inference operators — GQA decode 1.36x, MLA decode 1.21x, MLA prefill 1.50x, RMSNorm 1.12x — on NVIDIA B200; all four bars clear the 1.0x expert line." width="640" />
- <br/>
- <i>Geomean speedup over the FlashInfer <b>expert</b> baseline (the strongest kernel FlashInfer ships) on NVIDIA B200 — every op clears the expert. Full table in <a href="#results">Results</a>.</i>
- </p>
- ## News
- - 🔌 **[2026.08.17]** [**dsh-kernel-opt**](https://github.com/xieTwim/dsh-kernel-opt) is open-source — a [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (DSH) plugin that makes a kernel-optimization run watchable and steerable live; AKO4ALL's evaluator ships inside it as the built-in default.
- - 📄 **[2026.05.31]** The **[AKO tech report](https://tongminglaic.github.io/AKO/assets/ako-tech-report.pdf)** is now available.
- - 🚀 **[2026.05.31]** [**AKO4X**](https://github.com/TongmingLAIC/AKO4X) is now open-source — the closed-loop, campaign-based system behind our [MLSys 2026 competition](https://mlsys26.flashinfer.ai/) entry.
- - ✨ **[2026.05.31]** **AKO4ALL** is now a single drop-in [Claude Code](https://docs.anthropic.com/en/docs/claude-code) skill — invoke it in any working directory.
- - 🚀 **[2026.03.24]** AKO4ALL is released.
- **Table of Contents**
- - [What is AKO4ALL?](#what-is-ako4all)
- - [What You Provide](#what-you-provide)
- - [Install](#install)
- - [How to Use](#how-to-use)
- - [Requirements](#requirements)
- - [How It Works](#how-it-works)
- - [Results](#results)
- - [Agent Behavior](#agent-behavior)
- - [Permissions](#permissions)
- - [Repo Layout](#repo-layout)
- - [Example: SOL-ExecBench](#example-sol-execbench)
- - [Anti-Cheat](#anti-cheat)
- - [FAQ](#faq)
- - [Tech Report](#tech-report)
- - [Acknowledgments](#acknowledgments)
- ## What is AKO4ALL?
- **AKO4ALL is automated GPU kernel optimization, packaged as a single [Claude Code](https://docs.anthropic.com/en/docs/claude-code) skill.** Drop a kernel into your working directory, invoke the skill, and the agent bootstraps a workspace in place and iteratively rewrites the kernel for maximum performance — profile, edit, benchmark, repeat, until the kernel stops getting faster. The skill is a single `SKILL.md` protocol document; other coding agents can drive the same loop by following it directly.
- <p align="center">
- <img src="assets/hero.png" alt="A cartoon robot agent at a single workshop bench iterates on a glowing GPU kernel cube — with iteration logbook, profiler, timer, and commit stamp at hand, and a speedup chart on the chalkboard — illustrating AKO4ALL's single-session, drop-in kernel optimization loop." width="780" />
- </p>
- **AKO4ALL vs [AKO4X](https://github.com/TongmingLAIC/AKO4X)** — two sibling projects with different framings.
- - **Form:** AKO4ALL is a single Claude Code skill that runs in place in your working directory. AKO4X is a template repository that builds a fresh isolated child environment for each run.
- - **Unit of work:** AKO4ALL is *one continuous optimization run on one kernel* — resumable across chat sessions, but with no memory carried into the next run on the same kernel. AKO4X is a *multi-round campaign on one operator* — a persistent master agent spawns rounds back-to-back, and a per-operator archive accumulates each round's results so the Nth round is informed by all earlier rounds.
- - **Harness evolution:** AKO4ALL's protocol is static — you can supply behavior directives, but the skill itself does not change. AKO4X has an opt-in *harness co-evolution* mode in which the agent, after each round, can propose improvements to its own knowledge and tooling layer that the master gates and applies — no analogue here.
- Reach for AKO4ALL when you just want a kernel optimized; reach for AKO4X when you want infrastructure around the optimization.
- ## What You Provide
- A kernel and at least one set of test inputs are required. Everything else is optional.
- - **Kernel** (required) — The kernel to optimize. Can be a single file or a directory. Supports Triton, CUDA, C++, TileLang, CuTe DSL, Python, or any language that can be benchmarked.
- - **Reference implementation** (optional) — Used as the correctness golden. If absent, the original kernel is used.
- - **Inputs** (at least one set required) — In any form: hardcoded inside the kernel/reference, defined in your benchmark script, a separate Python file with `get_inputs()`, or raw data files (`.npz`, `.bin`, shape lists, etc.) that the agent wires up itself. The agent assembles whatever is provided into something the bench script can call.
- - **Benchmark script** (optional) — Your own benchmark script; the agent reads it to figure out how to run it. If no benchmark script is provided, the built-in [KernelBench](https://github.com/ScalingIntelligence/KernelBench) evaluator is used automatically (no setup needed beyond PyTorch).
- - **Knowledge** (optional) — Reference materials for the agent: algorithm descriptions, papers, design docs, or any background knowledge that helps inform the optimization.
- - **Hints** (optional) — Directives for the agent: optimization constraints, focus areas, and behavior controls (e.g., whether to allow web search).
- ## Install
- Clone the repo directly into Claude Code's skills directory:
- ```bash
- git clone https://github.com/TongmingLAIC/AKO4ALL.git ~/.claude/skills/ako4all
- ```
- Or, if you already have it cloned somewhere else, symlink it:
- ```bash
- ln -s /path/to/AKO4ALL ~/.claude/skills/ako4all
- ```
- ## How to Use
- ### Mode 1 — Interactive (default)
- Open Claude Code in a directory containing your kernel, then ask it to optimize the kernel. Files in the working directory are inspected automatically; you can also point at external paths. Optional constraints — iteration cap, language preference, dependency policy, etc. — included in the prompt will be merged into `HINTS.md`.
- The skill presents a resolved plan, asks if anything is genuinely ambiguous, and starts iterating. You can interrupt at any point to redirect.
- <details>
- <summary>Optional: pre-organize your workspace</summary>
- If you want the conventional layout the skill recognizes:
- ```
- <workspace>/
- ├── source/ # Kernel + optional reference and inputs
- │ ├── kernel.py
- │ ├── reference.py # Optional
- │ └── inputs.py / data.npz ... # Optional — input data, any format
- ├── bench/ # Your own benchmark script (optional)
- │ └── ...
- ├── knowledge/ # Reference materials (optional)
- └── HINTS.md # Agent directives (optional)
- ```
- These are conventions, not requirements. If your files are organized differently, just point Claude at them.
- </details>
- ### Mode 2 — Batch via subagents (advanced)
- For optimizing several related kernels in parallel, in Claude Code the main agent can spawn one subagent per kernel via the `Task` tool.
- > ⚠️ **Subagents can't ask back.** In Claude Code, subagents lack `AskUserQuestion`. The main agent composes each subagent's prompt from your instructions plus what's in each kernel's directory; any decision you don't pin down there will fall to `SKILL.md` defaults. So include the things you care about (per-kernel paths, optional reference / inputs locations, language or iteration-cap constraints, custom branch names) — and let the rest default.
- ## Requirements
- - A coding agent (e.g., [Claude Code](https://docs.anthropic.com/en/docs/claude-code))
- - NVIDIA GPU with CUDA-enabled PyTorch (only for the built-in evaluator; a remote-execution bench script may not need either locally)
- - Python **>= 3.10** for the built-in evaluator (or whatever your custom bench script requires)
- - NVIDIA Nsight Compute (`ncu`) — recommended; its version must match your CUDA / PyTorch CUDA build. If unavailable, the loop still proceeds (no profiling — the agent reasons from runtime stats instead).
- - Git
- > **Note:** During the loop the agent may run `pip install` / `apt install` etc. to fill in missing dependencies. A container or virtual environment is recommended for isolation; to forbid installs, add a directive to `HINTS.md` (see [Agent Behavior](#agent-behavior)) or restrict via [Permissions](#permissions).
- ## How It Works
- The agent creates an `opt/<kernel>` branch, copies your kernel to `solution/`, and generates `scripts/bench.sh` to wrap your benchmark. After verifying the baseline (and profiling it with `ncu` if available), it iterates: edit `solution/` → benchmark → log → commit. After 3 stagnant iterations it re-profiles, plans the next direction, and continues. Defaults like the stall threshold live in `HINTS.md`; iteration history lives in `ITERATIONS.md`. See [`SKILL.md`](SKILL.md) for the full protocol.
- ## Results
- <details>
- <summary><b>A drop-in skill that turns one prompt into an expert-beating kernel in under an hour — a clean sweep over FlashInfer's expert on all 4 inference ops.</b></summary>
- <br>
- Driven by **Claude Opus 4.8** (1M context, max thinking effort), on four FlashInfer-Bench inference operators AKO4ALL rewrote the reference into Triton kernels that **all beat FlashInfer's strongest _expert_ kernel** — geomean speedup over the FlashInfer expert on NVIDIA B200, all correctness-passing:
- | Operator | Speedup vs FlashInfer expert | Workloads |
- |---|---|---|
- | GQA paged decode | **1.36×** | 48 |
- | MLA paged decode | **1.21×** | 47 |
- | MLA paged prefill | **1.50×** | 38 |
- | RMSNorm h128 | **1.12×** | 14 |
- *All four operators we ran are shown here — we time-boxed this round to four kernels, so these are not the best four cherry-picked from a larger sweep.*
- **What one kernel takes to optimize** — a typical run, end-to-end from the initial prompt to the agent finishing:
- | End-to-end | Input | Output | Cache read | Cache write |
- |---|---|---|---|---|
- | ~55 min | ~15k | ~253k | ~17.6M | ~752k |
- A single drop-in skill — competitive-with-expert kernels in under an hour, from a single prompt.
- </details>
- ## Agent Behavior
- A few defaults worth knowing:
- - **Language switching** — the agent may rewrite your kernel in a different language (e.g., Triton → CUDA) to chase performance.
- - **Web search** — enabled; the agent searches for optimization ideas after consecutive stagnant iterations.
- - **Profiling** — `ncu` runs on the baseline before iter-1 and again after 3 stagnant iterations.
- To adjust any of this — or to add your own directives (optimization constraints, dependency policies, iteration caps, ...) — edit `HINTS.md` in your workspace. The file is self-documenting.
- ## Permissions
- The optimization loop involves running shell commands (compiling, benchmarking, profiling). By default, most coding agents prompt for approval on each command. To run fully unattended, grant the necessary permissions through your agent's configuration.
- For Claude Code, the simplest option is to bypass all permission checks:
- ```bash
- claude --dangerously-skip-permissions
- Note: Starts a coding agent with its permission prompts off
- Line 179 on GitHub
- ```
- <details>
- <summary>Granular control via settings file</summary>
- Create `.claude/settings.local.json`:
- ```json
- {
- "permissions": {
- "allow": [
- "Bash(*)", "Read(*)", "Write(*)", "Edit(*)",
- "Glob(*)", "Grep(*)", "Agent(*)",
- "WebFetch(*)", "WebSearch(*)"
- ]
- }
- }
- ```
- </details>
- For other agents, consult their documentation on permission / auto-approve settings.
- ## Repo Layout
- ```
- AKO4ALL/
- ├── SKILL.md # Entry point — loaded by Claude Code when the skill triggers
- ├── HINTS.md # Scaffold: agent behavior defaults
- ├── ITERATIONS.md # Scaffold: iteration log template
- ├── bench-wrapper.sh # Scaffold: bench script template
- └── bench/kernelbench/ # Scaffold: built-in KernelBench evaluator
- ```
- Scaffold files are copied into each workspace on first bootstrap, never overwriting existing files. `README.md`, `LICENSE`, and `assets/` are project metadata — not installed anywhere.
- ## Example: SOL-ExecBench
- [SOL-ExecBench](https://github.com/NVIDIA/SOL-ExecBench) contains 235 real-world DL kernel problems from NVIDIA. AKO4ALL can optimize any of them by pointing at the problem directory directly — no copying into a local workspace.
- 1. Install the skill (see [Install](#install)) and clone SOL-ExecBench separately. Set up its environment and `ncu` per its own instructions.
- 2. Activate the SOL-ExecBench environment and open Claude Code in any working directory:
- ```bash
- claude
- ```
- 3. Give it a prompt (replace the paths and iteration cap):
- ```
- Optimize the kernel at <sol-bench>/data/benchmark/L1/001_attention_softmax_dropout_value_matmul_backward.
- Bench with SOL-ExecBench at <sol-bench>; its dependencies are already installed.
- Optimize for up to 30 iterations. Stop early only if all viable approaches are exhausted.
- ```
- The skill resolves the external path, wraps SOL-ExecBench's bench command as `scripts/bench.sh`, and starts iterating.
- ## Anti-Cheat
- Three layers of protection against the agent gaming the metric instead of achieving real speedup:
- 1. **Agent rules.** The skill instructs the agent to pursue genuine latency reduction — no CUDA stream injection, no timing manipulation, no returning uninitialized results. Full list in [`SKILL.md`](SKILL.md) "Gotchas".
- 2. **Evaluator checks** (built-in KernelBench). Flags suspicious >10× speedups for review; strips any input-generating code (e.g., `get_inputs`) from the solution file before evaluation — so the solution can't choose what it's tested against; times with **fresh input values per trial by default** (both solution and reference; `--no-fresh-inputs` restores the historical reused regime — the two regimes' numbers are different quantities, never compare across them), so no trial can be served from a value cache; and, after the timed trials, re-randomizes the inputs **in place** and requires the solution to still track the reference (mutation sentinel) — a solution that keys on input identity and replays a stored output fails instead of banking a cache-hit timing. On failure the output carries a `DEVIATION:` line, so a numerics near miss and a gross error are distinguishable.
- 3. **Stricter enforcement** (optional). Provide a custom bench script with static analysis — e.g., KernelBench's [`kernel_static_checker.py`](https://github.com/ScalingIntelligence/KernelBench) — to reject solutions that contain disallowed patterns before they are even timed.
- ## FAQ
- **1. Can I use AKO4ALL for a whole benchmark suite?**
- For ad-hoc optimization of several kernels at once, see [Mode 2](#mode-2--batch-via-subagents-advanced). For campaign-grade multi-round optimization — cross-run memory, an archive of variants feeding the next round, and master/sub agent separation — see [AKO4X](https://github.com/TongmingLAIC/AKO4X) (default benchmark: flashinfer-bench, swappable via adapter).
- **2. My agent isn't trying hard enough — keeps tuning configs instead of rewriting.**
- Agent laziness is a known failure mode (staying in PyTorch, only tuning configurations, skipping profiling). Intervene with specific guidance, or add directives to `HINTS.md` ahead of time. The point of the loop is to *rewrite* the kernel — switch languages (Triton → CUDA, etc.) when it helps.
- **3. Which model should I use?**
- Model capability strongly influences optimization quality. We recommend [Claude Opus](https://docs.anthropic.com/en/docs/about-claude/models).
- **4. How do I cap iterations?**
- By default there is no cap — the agent decides when to stop on its own. To enforce a limit, add a directive to `HINTS.md`, or include it in your prompt (e.g., `Optimize for up to 30 iterations. Stop early only if all viable approaches are exhausted.`).
- **5. My bench script uses a remote service (e.g., Modal). Does that work?**
- Yes. As long as your bench script runs from the command line and prints results to stdout. The local machine doesn't need a GPU / CUDA / PyTorch in that case — just whatever your wrapper script imports.
- **6. Can I intervene during optimization?**
- Yes. Interrupt to redirect, or to manually edit files in `solution/` — then tell the agent to continue.
- ## Tech Report
- The **[AKO tech report](https://tongminglaic.github.io/AKO/assets/ako-tech-report.pdf)** is now available.
- ## Citation
- If you find AKO useful, please cite:
- ```bibtex
- @misc{ako2026,
- title = {{AKO}: Agentic Kernel Optimization},
- author = {Shuxiao Xie and Shuyang Xie and Dezhi Ran and Wei Yang and Tao Xie},
- year = {2026},
- howpublished = {\url{https://tongminglaic.github.io/AKO}},
- note = {Technical report}
- }
- ```
- ## Acknowledgments
- We would like to thank the following open-source projects that inspired and supported the development of AKO:
- - [KernelBench](https://github.com/ScalingIntelligence/KernelBench) — for providing the benchmark and evaluation format used by AKO4ALL's built-in evaluator.
- - [autoresearch](https://github.com/karpathy/autoresearch) and [autokernel](https://github.com/RightNow-AI/autokernel) — AKO's design was inspired by their work on autonomous optimization loops.
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.