Skip to content
Back to search

inference-optimisation-skill

Agent skill for hill-climbing local LLM serving speed (decode, speculative decoding, prefill, KV, startup, TTFT) on any hardware and quantization

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked yesterday
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/sybil-solutions--inference-optimisation-skill/badge.svg)](/listings/sybil-solutions--inference-optimisation-skill)

HTML

<a href="/listings/sybil-solutions--inference-optimisation-skill"><img src="/api/listings/sybil-solutions--inference-optimisation-skill/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Claude Code (user-level):

git clone https://github.com/sybil-solutions/inference-optimisation-skill
mkdir -p ~/.claude/skills
cp -r inference-optimisation-skill/skills/inference-optimisation ~/.claude/skills/

Then ask, for example:

- "hill-climb decode and prefill for the model on :8000 under a 40 GiB cap" - "what's the theoretical max tok/s for this model on this machine?"

Other agents: point them at `skills/inference-optimisation/SKILL.md`, or paste [`PROMPT.md`](PROMPT.md) as the task prompt.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 151 lines
  1. # inference-optimisation
  2. An agent skill for **hill-climbing the speed of a local LLM server** on any hardware and any quantization. It covers:
  3. - single-user and concurrent decode
  4. - speculative decoding
  5. - prefill
  6. - KV cache size
  7. - startup time and time to first token
  8. It works under a memory cap, and every result has to survive a full-context stability check.
  9. It encodes how we actually got these speedups. It is not a list of flags. The workflow:
  10. 1. Compute the theoretical ceiling.
  11. 2. Change one thing at a time.
  12. 3. Measure against interleaved controls.
  13. 4. Log every result, including the failures.
  14. 5. Gate on long-context retrieval.
  15. 6. Qualify a clean, pinned build before publishing.
  16. It works with Claude Code and any agent that loads `SKILL.md`-style skills. Any model can also follow it from [`PROMPT.md`](PROMPT.md).
  17. ## What we've been able to accomplish
  18. Results from the hill-climbs this skill was distilled from. Every number was measured on the hardware named; see the linked sources for methods and raw data.
  19. | Hardware | Model / quant / engine | Before → after |
  20. |---|---|---|
  21. | **Intel Arc B390 iGPU** (laptop, 64 GB unified LPDDR5X, ~115 GB/s) | Ornith-1.5-35B-A3B, EXL3 4 bpw, SGLang XPU + MTP | Decode, 1 user: 43.6 → **53 tok/s** (median)<br>Decode, 4 users: 84.7 → **103 tok/s**<br>Prefill 32K: 1,037 → **1,660 tok/s**<br>Prefill 256K: 297 → **628 tok/s** (2.1×)<br>KV cache: 264K tokens → **4 × 256K**<br>Startup: 113 → **43 s**<br>Registry Lab gates: decode 28.6 → **41.0**, prefill 255 → **667** |
  22. | **Intel Arc Pro B70** | Qwen3.8-27B, EXL3 4 bpw, vLLM + custom XPU kernels | Decode: **91.2 tok/s** at 1 user, **365** at 16 (prose, thinking on)<br>Same card with llama.cpp Q4_K_M: 25.0<br>Prefill 4K/32K/128K/254K: 1,654/1,434/928/650 → **2,421/2,259/1,415/911**<br>Agent-session replay: 193 → **147 s** |
  23. | **Intel Arc Pro B70** | Same model, SGLang + the same kernels | Decode, 1 user: **82.6 tok/s**, vs vLLM 75.7 on the same card and day |
  24. | **RTX 3090** | Qwen3.8-27B EXL3 3 bpw + MTP | **96.2 tok/s** prose / **141.1** code, 1 user |
  25. | **RTX 3090** | Qwen3.6-35B-A3B EXL3 3 bpw + MTP | **251.7 / 357.9 tok/s**, 1 user |
  26. | **RTX 3090** | Flash-Next EXL3, 256K context | MTP: 45.5 → **65.6 tok/s** |
  27. | **4× H200 NVL** | DeepSeek-V4.1-Flash, vLLM + speculative decoding | Crashed under load → **229.1 tok/s** decode, **17,546 tok/s** prefill; all six gates pass |
  28. Sources:
  29. - [local-ai-registry](https://github.com/sybil-solutions/local-ai-registry):
  30. - [Ornith B390 recipe](https://github.com/sybil-solutions/local-ai-registry/tree/main/data/sources/ornith-b390-exl3), with the full hill-climb log in [PR #453](https://github.com/sybil-solutions/local-ai-registry/pull/453)
  31. - the RTX 3090 and H200 recipes
  32. - [trellis-serve](https://github.com/sybil-solutions/trellis-serve):
  33. - `xpu/README.md`, `xpu/docs/PROGRESS.md` and `xpu/docs/SGLANG.md` for B70
  34. - `cuda/README.md` for the RTX 3090
  35. - [qwen-serve](https://github.com/sybil-solutions/qwen-serve)
  36. ### Reading the B390 result
  37. That run is the one this skill grew out of. Its ceiling calculation said:
  38. - **Decode:** 100 tok/s for one user was only reachable with two useful speculative drafts. This model's draft head accepts ~14% at depth 2, so the practical target was ~88–97 tok/s. We reached ~55% of it; the rest is GEMV kernel efficiency.
  39. - **Prefill at 256K:** we reached 628 tok/s against a ~670–684 tok/s ceiling.
  40. The skill makes the agent compute this up front, so it can tell you early when a target is above the physics.
  41. ## How it works
  42. ```
  43. ┌────────────────────────────────────────────────────────────────┐
  44. measure HW ─▶ │ THEORY.md: bytes/token ÷ bandwidth, FLOPs/token ÷ 50% peak, KV │
  45. └────────────────────────────────────────────────────────────────┘
  46. │
  47. baseline ─▶ profile ─▶ pick lever (ceiling gap × likelihood) ─▶ configs/NEW.json
  48. ▲ │
  49. │ queue.sh ctl NEW ctl (serialized, AC-only,
  50. │ memory watchdog, ≥6 samples, medians)
  51. │ │
  52. PLAN.md + ledger.jsonl ◀── accept only if it beats BOTH neighbouring controls ──┘
  53. │ and regresses nothing by >3%
  54. ▼
  55. long-context needle gate ─▶ concurrent full-KV soak ─▶ plateau? ─▶ pinned clean build
  56. ─▶ acceptance gates ─▶ publish
  57. ```
  58. The skill has three parts:
  59. **[`SKILL.md`](skills/inference-optimisation/SKILL.md): the procedure.**
  60. - Inventory, ceilings, ground rules, the loop and lever order.
  61. - When to stop, how to qualify and publish, and how to report.
  62. **[`references/`](skills/inference-optimisation/references): loaded only when needed.**
  63. - [`ceilings.md`](skills/inference-optimisation/references/ceilings.md): how to compute decode, prefill, TTFT, startup and KV maxima, with a worked example.
  64. - [`methodology.md`](skills/inference-optimisation/references/methodology.md): measurement traps. Chunk-counted tokens, mis-sized prompts, acceptance counters, drift, unified-memory accounting.
  65. - [`levers.md`](skills/inference-optimisation/references/levers.md): every lever with measured evidence. It is tagged general vs stack-specific.
  66. - [`gotchas.md`](skills/inference-optimisation/references/gotchas.md): known traps in vLLM, SGLang, llama.cpp and ExLlama, and on Intel, NVIDIA, Apple and AMD hardware.
  67. - [`publishing.md`](skills/inference-optimisation/references/publishing.md): pinning, clean builds, the six acceptance gates, proving the build matches the best, results cards.
  68. **[`scripts/`](skills/inference-optimisation/scripts): a small, engine-agnostic harness.** It is standard-library Python plus bash.
  69. - `bench.py` benchmarks any OpenAI-compatible server. It measures:
  70. - startup and short-prompt TTFT;
  71. - c1 decode (median of N) and cN aggregate decode;
  72. - cold prefill at several lengths;
  73. - long-context needle retrieval;
  74. - peak memory, with a cap watchdog.
  75. It flags looping output, failed requests and runs on battery. Token counts come from server usage stats, never from streamed chunks, and everything lands in `ledger.jsonl`.
  76. - `ceiling.py` computes the theoretical ceilings from a JSON of measured bandwidth and FLOPs plus component sizes.
  77. - `queue.sh` is a serialized, detachable run queue, for interleaving control/candidate/control runs.
  78. - `mem.sh` reports a container's real footprint. That includes driver residency on unified-memory GPUs, which cgroups don't see.
  79. ## Install
  80. Claude Code (user-level):
  81. ```bash
  82. git clone https://github.com/sybil-solutions/inference-optimisation-skill
  83. mkdir -p ~/.claude/skills
  84. cp -r inference-optimisation-skill/skills/inference-optimisation ~/.claude/skills/
  85. ```
  86. Then ask, for example:
  87. - "hill-climb decode and prefill for the model on :8000 under a 40 GiB cap"
  88. - "what's the theoretical max tok/s for this model on this machine?"
  89. Other agents: point them at `skills/inference-optimisation/SKILL.md`, or paste [`PROMPT.md`](PROMPT.md) as the task prompt.
  90. ## Quick start without an agent
  91. ```bash
  92. S=skills/inference-optimisation
  93. python3 $S/scripts/ceiling.py $S/examples/ceiling-ornith-b390.json # ceilings
  94. mkdir -p configs && cp $S/examples/config-existing-server.json configs/base.json # edit base_url/model
  95. python3 $S/scripts/bench.py run configs/base.json --long 131072 # one run
  96. bash $S/scripts/queue.sh base cand base # interleaved A/B
  97. python3 $S/scripts/bench.py show # ledger table
  98. ```
  99. ## Principles that carried the most weight
  100. - **Ceilings first.** If you don't know the maximum, you can't tell whether 53 tok/s is great or bad, or whether "100 tok/s" is possible.
  101. - **Noise is ±5–10%.** Most single-run "wins" are drift. Interleave controls and use medians.
  102. - **Measure what matters:**
  103. - real tokens, not stream chunks;
  104. - real prompt lengths, from the tokenizer;
  105. - real memory, including GPU residency on unified memory.
  106. - **Speculative decoding is about per-step cost, not acceptance.**
  107. - Prune the draft head's vocabulary.
  108. - Sweep draft length per concurrency and context.
  109. - Keep sampling on the GPU.
  110. - **Prove full-context stability** with exact-match needles and a concurrent full-KV soak before calling anything "best".
  111. - **Write down failures.** Most of the value in `PLAN.md` is the list of things that did *not* work and why.
  112. ## Contributing
  113. PRs with new levers, gotchas or results are welcome. Each needs the hardware, model, quantization, engine version, before → after numbers, how they were measured, and a link to raw data.
  114. ## License
  115. MIT

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.