Skip to content
Back to search

cuda-optimized-skill

A CUDA kernel optimization toolkit for validation, benchmarking, Nsight Compute profiling, bottleneck analysis, and iterative tuning. It helps improve custom GPU operators with reproducible workflows and evidence-based performance comparison.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/kernelflow-ops--cuda-optimized-skill/badge.svg)](/listings/kernelflow-ops--cuda-optimized-skill)

HTML

<a href="/listings/kernelflow-ops--cuda-optimized-skill"><img src="/api/listings/kernelflow-ops--cuda-optimized-skill/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 218 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # cuda-kernel-optimizer
  2. **English** | [简体中文](README.zh-CN.md)
  3. A Claude skill that iteratively optimizes a CUDA / CUTLASS / Triton kernel against a Python reference, using `nsight-compute` (`ncu`) as the source of evidence for each optimization decision.
  4. This is a **skill package**, not a standalone tool. Claude reads `SKILL.md` and drives the loop. The scripts under `scripts/` handle the deterministic parts (environment detection, profiling, benchmarking, state).
  5. ---
  6. ![alt text](asset/v2_en_arch.png)
  7. ## Usage
  8. ```text
  9. Use this prompt in the agent:
  10. @cuda-kernel-optimizer use this skill to optimize "the operator you want to optimize" for N iterations.
  11. ```
  12. ## What's new in V2
  13. V2 upgrades the loop from "try-and-log" into "try–attribute–verify–learn". Four mechanisms are added on top of V1; everything below reflects V2 behavior:
  14. - **Roofline-driven axis budget** — instead of V1's fixed 1-method-per-axis, V2 computes per-iteration compute/memory/latency gaps (Δc, Δm, Δl) and splits the 3-method budget proportionally (per-axis cap = 2). When all three gaps fall below 0.15 the loop early-stops with `near_peak: true`.
  15. - **Branch-and-Select exploration** — each iteration generates K branch candidates (default K=4) sharing the same methods but varying tile size, pipeline stages, warp count, and implementation variants. The fastest correct branch wins as champion; the rest are archived in `frontier`.
  16. - **Ablation-based attribution** — after the champion is picked, each method is ablated one at a time. `attribution(m) = ms_without_m − ms_champion` gives a per-method causal contribution instead of a single packed verdict.
  17. - **SASS instruction-level verification** — `cuobjdump --dump-sass` is grepped against a signature table (`sass_signatures.json`) to confirm each claimed optimization actually appears in the compiled machine code.
  18. These together change method classification from two buckets (effective / ineffective) to three: `effective_methods` (SASS ✓ and attribution > noise), `ineffective_methods` (SASS ✓ but attribution ≤ noise), and `implementation_failed_methods` (SASS ✗).
  19. ## What you need
  20. On the host where Claude runs:
  21. - A CUDA GPU with working drivers (`nvidia-smi` works)
  22. - `nvcc` for CUDA / CUTLASS backends
  23. - `ncu` with permission to read perf counters
  24. - `cuobjdump` (ships with the CUDA toolkit) for SASS verification
  25. - Python 3.10+ with `torch` (CUDA build), `triton` if you want the Triton backend
  26. - For CUTLASS kernels: `$CUTLASS_PATH` or `$CUTLASS_INCLUDE_DIR` pointing at a tree with both `cutlass/` and `cute/` headers
  27. `benchmark.py` (the generic operator benchmark driver) is bundled at `scripts/benchmark.py` — no separate installation needed.
  28. The strict hardware gate searches `PATH`, CUDA roots, and common Nsight locations. If GPU runtime or any required tool remains unavailable after three probes, optimization stops before candidate generation.
  29. ### `ncu` permission gotcha
  30. On most cloud and container setups, profiling-counter access is disabled. You'll see it as `can_read_counters: false` in `env.json`. Fixes (pick one):
  31. - Run the host as root, or
  32. - Add `options nvidia NVreg_RestrictProfilingToAdminUsers=0` to `/etc/modprobe.d/nvidia.conf` and reboot, or
  33. - For docker: `--cap-add=SYS_ADMIN` (Nsight docs recommend this)
  34. ## What you give Claude
  35. 1. **Baseline kernel file** — `gemm.cu` (CUDA/CUTLASS) or `gemm.py` (Triton)
  36. 2. **Reference file** — `ref.py` exposing `reference(**kwargs)` and optional `atol` / `rtol`
  37. 3. **Dims** — the scalar args the signature takes (e.g. `M=4096 N=4096 K=4096`)
  38. 4. **Path to `benchmark.py`** — already bundled under `scripts/benchmark.py`; `orchestrate.py` defaults to it. Pass `--benchmark <path>` only if you have a custom version.
  39. 5. Optional: iteration count `N` (default 3), `ncu_num` per-axis top-K (default 5), noise threshold (default 2%), **branches per iteration `K` (default 4, via `--branches`)**
  40. ## What you get back
  41. A sibling directory of your baseline, `run_YYYYMMDD_HHMMSS/`, containing:
  42. ```text
  43. run_YYYYMMDD_HHMMSS/
  44. ├── state.json # global state, re-readable across sessions
  45. │ # V2 adds: branches, implementation_failed_methods,
  46. │ # roofline_history, frontier
  47. ├── env.json # GPU / nvcc / ncu / CUTLASS snapshot
  48. ├── baseline/
  49. │ ├── <baseline> # copied verbatim
  50. │ └── bench.json # seed timing + correctness
  51. ├── iterv1/
  52. │ ├── roofline.json # Δc / Δm / Δl + per-axis budget allocation
  53. │ ├── methods.json # methods picked under the budget (trigger_strength included)
  54. │ ├── analysis.md # ncu metrics + CoT + risk notes
  55. │ ├── best_input.ncu-rep # profile of what went IN
  56. │ ├── branches/ # K branch candidates (same methods, different hyperparams)
  57. │ │ ├── b0/kernel.{cu,py} + bench.json
  58. │ │ ├── b1/…
  59. │ │ └── …
  60. │ ├── kernel.{cu,py} # champion kernel (fastest correct branch)
  61. │ ├── kernel.ncu-rep # profile of the champion
  62. │ ├── ncu_top.json # top-K metrics per axis (what Claude sees)
  63. │ ├── sass_check.json # per-method SASS signature verification
  64. │ ├── ablations/ # leave-one-out ablation runs
  65. │ │ ├── no_<method_a>/kernel.{cu,py} + bench.json
  66. │ │ └── …
  67. │ ├── attribution.json # per-method causal contribution (ms)
  68. │ └── bench.json
  69. ├── iterv2/ …
  70. ├── iterv3/ …
  71. └── summary.md # headline speedup, timeline, bottleneck drift, retrospective
  72. ```
  73. ## Manual invocation
  74. You don't need to drive the loop by hand — that's Claude's job — but for debugging the skill itself:
  75. ```bash
  76. # 0 + 0b + 1 + 2 + 3a-for-iter1
  77. python scripts/orchestrate.py setup \
  78. --baseline ./gemm.cu \
  79. --ref ./ref.py \
  80. --iterations 3 \
  81. --ncu-num 5 \
  82. --branches 4 \
  83. --dims '{"M":4096,"N":4096,"K":4096}'
  84. # --benchmark defaults to scripts/benchmark.py (bundled)
  85. # --- (Claude writes iterv1/kernel.cu + iterv1/methods.json + iterv1/analysis.md
  86. # + K branch candidates under iterv1/branches/) ---
  87. # 3d + 3f + 3a-for-iter2 for iter 1
  88. # close-iter now also runs: branch selection → SASS check → ablation → state update
  89. python scripts/orchestrate.py close-iter \
  90. --run-dir run_20260418_143022 \
  91. --iter 1
  92. # --benchmark defaults to scripts/benchmark.py (bundled)
  93. # (repeat code-gen + close-iter for iter 2 and iter 3)
  94. # 4
  95. python scripts/orchestrate.py finalize --run-dir run_20260418_143022
  96. ```
  97. Each script is independently invocable (`--help` on any of them); `orchestrate.py` is just a convenience wrapper.
  98. ## Repo layout
  99. ```text
  100. cuda-kernel-optimizer/
  101. ├── SKILL.md # the skill — Claude reads this
  102. ├── README.md # you are here
  103. ├── scripts/
  104. │ ├── benchmark.py # bundled benchmark driver (from project)
  105. │ ├── check_env.py # detect GPU / nvcc / ncu / cuobjdump / CUTLASS / libs
  106. │ ├── hardware_gate.py # strict tool discovery + CUDA runtime gate (3 attempts)
  107. │ ├── strict_validation.py # benchmark/NCU predicates + durable stop state
  108. │ ├── preflight.py # validate baseline + ref contract
  109. │ ├── state.py # the ONLY writer of state.json
  110. │ ├── validate_methods.py # priority-compliance gate (called by state.py)
  111. │ ├── run_iteration.py # calls benchmark.py, captures results
  112. │ ├── profile_ncu.py # runs ncu, extracts top-K per axis
  113. │ ├── roofline.py # [V2] compute Δc/Δm/Δl, allocate axis budget, near_peak check
  114. │ ├── branch_explore.py # [V2] compile + benchmark K branches, elect champion, update frontier
  115. │ ├── ablate.py # [V2] leave-one-out ablation, emit per-method attribution
  116. │ ├── sass_check.py # [V2] cuobjdump → grep signatures → per-method SASS verdict
  117. │ ├── summarize.py # renders summary.md (V2: includes bottleneck drift table)
  118. │ └── orchestrate.py # end-to-end CLI (setup/close-iter/finalize)
  119. ├── references/
  120. │ ├── ncu_metrics_guide.md # bottleneck → optimization mapping
  121. │ ├── optimization_catalog.md # priority-ordered catalog (Claude reads)
  122. │ ├── method_registry.json # machine-readable mirror (validator reads)
  123. │ └── sass_signatures.json # [V2] method → expected SASS instruction signatures
  124. ├── templates/
  125. │ ├── iteration_report.md # analysis.md skeleton Claude fills in
  126. │ └── methods.schema.json # schema for methods.json (V2: adds trigger_strength)
  127. └── examples/
  128. └── walkthrough.md # annotated example session
  129. ```
  130. ## How Claude uses this
  131. When a user says "optimize `gemm.cu`", Claude:
  132. 1. reads `SKILL.md`
  133. 2. calls `orchestrate.py setup` (which runs env check → preflight → init → seed baseline → first profile)
  134. 3. reads `iterv1/ncu_top.json` and the current best kernel source
  135. 4. **runs `roofline.py` to get Δc / Δm / Δl and the per-axis method budget (total = 3, per-axis cap = 2); if `near_peak: true`, the loop ends here**
  136. 5. consults `references/optimization_catalog.md` + `references/ncu_metrics_guide.md`
  137. 6. picks methods **under the axis budget** (budget-aware scan: skip axis if budget=0, pick top-N by `trigger_strength` if budget=2), writes them + reasoning to `iterv1/methods.json` and `iterv1/analysis.md`
  138. 7. writes **K branch candidates** to `iterv1/branches/b{0..K-1}/kernel.<ext>` — same methods, different hyperparameters (tile / stages / warps / impl variants)
  139. 8. calls `orchestrate.py close-iter --iter 1`, which internally:
  140. - runs `branch_explore.py` → compiles + benchmarks all branches, elects the fastest correct one as champion (copied to `iterv1/kernel.<ext>`), archives the rest in `frontier`
  141. - profiles the champion with `ncu` → `iterv1/kernel.ncu-rep`
  142. - runs `sass_check.py` → `iterv1/sass_check.json`
  143. - runs `ablate.py` → `iterv1/attribution.json`
  144. - updates state: each method lands in one of `effective_methods` / `ineffective_methods` / `implementation_failed_methods` based on SASS ✓/✗ × attribution > noise
  145. 9. if all K branches fail compilation, correctness, or stable timing: records `stop.json` and stops without generating later iterations
  146. 10. on success: `best_file` advances if faster; `roofline_history` is appended
  147. 11. loops back to step 3 for the next iteration
  148. 12. calls `orchestrate.py finalize` and writes a retrospective into `summary.md` — including the bottleneck drift table sourced from `roofline_history`
  149. See `examples/walkthrough.md` for a full example and `SKILL.md` for the formal procedure.
  150. ## Limits and honest caveats
  151. - **Ceiling**: if your reference is already cuBLAS / cuDNN / cuBLASLt, meaningful wins require algorithmic changes (split-K, stream-K, fused epilogues, mixed precision) that Claude may or may not find in a 3-iteration budget. Large speedups are easier when the baseline is hand-rolled.
  152. - **Noise**: kernels running under ~50 μs are dominated by launch overhead. The skill's default 2% noise threshold helps, but if your dims are tiny, raise `--repeat` or the dimensions. Ablation attribution uses the same threshold — sub-noise contributions are classified as `ineffective_methods`.
  153. - **Triton + `@triton.autotune`**: autotuning under `ncu` is slow and can time out. Either pre-bake a single config before profiling, or set `--launch-count 1` and increase warmup.
  154. - **ncu CSV column names**: older `ncu` (< 2022.1) emits `"Metric Value"` with different capitalization/units; `profile_ncu.py` is tolerant but if you see all zeros check the `.ncu.log` file in the iteration directory.
  155. - **Branch cost**: with K=4 and ablation, each iteration compiles up to K + (num_methods) kernels. On a fresh build this can be slow; lower `--branches` if wall-clock matters more than exploration.
  156. - **SASS signatures are heuristic**: `sass_signatures.json` greps for instruction patterns, not full semantic equivalence. A method can pass the grep but still be implemented suboptimally — attribution is what catches that.
  157. - **Retries are bounded**: hardware/tool discovery is attempted at most three times. Correctness, stable-timing, or NCU failure stops the run; later iterations are not generated.
  158. ## Example result
  159. Using the Batch Normalization problem from Tensara as an example, this project demonstrates a substantial performance improvement from a baseline implementation to an optimized kernel. After submission to the A100-80GB environment, the solution passed 4/4 test cases successfully. The average runtime dropped from 82.94 ms to 439.13 μs, while throughput increased dramatically from 2.52 GFLOPS to 476.20 GFLOPS. It is worth noting that most development and tuning were carried out locally on an RTX 3060, so local measurements cannot fully reflect the upper-bound performance achievable on an A100. Therefore, the final benchmark results should be based on the platform’s A100 evaluation, which better highlights the impact of careful kernel optimization and implementation details.
  160. ![alt text](asset/Tensara_baseline.png)
  161. ![alt text](asset/Tensara_best.png)
  162. ## License / attribution
  163. This skill is independent of and does not redistribute CUTLASS, Triton, or Nsight Compute. You need to install those separately.
  164. ## Star History
  165. <a href="https://www.star-history.com/?repos=KernelFlow-ops%2Fcuda-optimized-skill&type=date&legend=top-left">
  166. <picture>
  167. <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=KernelFlow-ops/cuda-optimized-skill&type=date&theme=dark&legend=top-left" />
  168. <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=KernelFlow-ops/cuda-optimized-skill&type=date&legend=top-left" />
  169. <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=KernelFlow-ops/cuda-optimized-skill&type=date&legend=top-left" />
  170. </picture>
  171. </a>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.