Skip to content
Back to search

fw-ai-cookbook

MarketplaceWorks with Claude CodeGitHubIncomplete

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026. Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/fw-ai-external/cookbook with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/fw-ai-external/cookbook'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/fw-ai--cookbook/badge.svg)](/listings/fw-ai--cookbook)

HTML

<a href="/listings/fw-ai--cookbook"><img src="/api/listings/fw-ai--cookbook/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Use Python 3.11 or newer. Install the released recipes and utilities from PyPI:

python -m pip install --upgrade fireworks-training-cookbook

Python imports remain `training.*`. The cookbook uses calendar versions and is released separately from the `fireworks-ai` SDK. See the [installation and migration guide](https://docs.fireworks.ai/fine-tuning/training-api/cookbook/overview#installation) for pinned releases and updating an existing checkout.

To edit recipes or run examples from a source checkout:

git clone https://github.com/fw-ai-external/cookbook.git
cd cookbook/training
uv venv --python 3.12 && source .venv/bin/activate
uv pip install -e .

See [`training/README.md`](./training/README.md) for recipes and examples.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 14 lines
  1. {
  2. "name": "fw-ai-cookbook",
  3. "owner": { "name": "Fireworks AI" },
  4. "metadata": {
  5. "description": "Fireworks training from coding agents: managed jobs and Training API cookbook workflows."
  6. },
  7. "plugins": [
  8. {
  9. "name": "fireworks-training",
  10. "source": "./",
  11. "description": "Skills for research, configuration, debugging, and migration of Tinker training loops to Fireworks."
  12. }
  13. ]
  14. }
README.md · README · 110 lines
  1. # Fireworks AI Cookbook
  2. Training loops and worked examples for RL (GRPO, IGPO, SAO/PPO), preference
  3. optimization (DPO, ORPO), SFT, distillation and embedding fine-tuning on
  4. [Fireworks](https://fireworks.ai). Full docs: [Training API](https://docs.fireworks.ai/fine-tuning/training-api/introduction).
  5. ## Quick Start
  6. Use Python 3.11 or newer. Install the released recipes and utilities from PyPI:
  7. ```bash
  8. python -m pip install --upgrade fireworks-training-cookbook
  9. ```
  10. Python imports remain `training.*`. The cookbook uses calendar versions and is
  11. released separately from the `fireworks-ai` SDK. See the
  12. [installation and migration guide](https://docs.fireworks.ai/fine-tuning/training-api/cookbook/overview#installation)
  13. for pinned releases and updating an existing checkout.
  14. To edit recipes or run examples from a source checkout:
  15. ```bash
  16. git clone https://github.com/fw-ai-external/cookbook.git
  17. cd cookbook/training
  18. uv venv --python 3.12 && source .venv/bin/activate
  19. uv pip install -e .
  20. ```
  21. See [`training/README.md`](./training/README.md) for recipes and examples.
  22. ## For AI Agents
  23. These skills bring Fireworks training know-how into compatible AI agents through
  24. progressive disclosure. Each entry point loads only the workflow guidance
  25. needed for the task, then follows linked Fireworks documentation and runnable
  26. cookbook examples when deeper detail is needed. The skill set provides **research**, **configure**, **debug**, and
  27. **migrate-tinker** workflows.
  28. | Skill | What it does | Try |
  29. |---|---|---|
  30. | [Research](skills/research/SKILL.md) | Helps decide whether training is the right intervention. It gathers the task, data, evaluation criteria, and constraints, then recommends a method and the closest runnable cookbook entry. It does not launch training. | *"Which cookbook entry fits prompt routing to small vs big models?"* |
  31. | [Configure](skills/configure/SKILL.md) | Turns a training goal into an executable plan for managed SFT, DPO, ORPO, or RFT, as well as serverless or dedicated Training API setups. It validates inputs, estimates cost, and supports running, monitoring, evaluation, deployment, resume, and teardown. It shows the complete plan and asks for approval before spend or mutation. | *"SFT qwen3-8b on my JSONL. Show the plan, but do not start yet."* |
  32. | [Debug](skills/debug/SKILL.md) | Diagnoses stuck, failed, slow, or low-quality training runs. It gathers runtime evidence, classifies the failure, suggests the safest next action, and does not retry or mutate resources without approval. | *"My job is stuck RUNNING at 0%."* |
  33. | [Migrate Tinker](skills/migrate-tinker/SKILL.md) | Ports a Tinker SDK or cookbook loop to native Fireworks clients, data types, sampling, checkpoints, and cleanup while preserving the experiment. | *"Migrate this Tinker loop to Fireworks, preserving its loss and token masks."* |
  34. ### Claude Code
  35. ```bash
  36. claude plugin marketplace add fw-ai-external/cookbook
  37. claude plugin install fireworks-training@fw-ai-cookbook
  38. ```
  39. ### Cursor
  40. ```bash
  41. npx --yes skills add fw-ai-external/cookbook -g \
  42. -s fireworks-training -s research -s configure -s debug -s migrate-tinker -a cursor -y
  43. ```
  44. ### Codex
  45. ```bash
  46. npx --yes skills add fw-ai-external/cookbook -g \
  47. -s fireworks-training -s research -s configure -s debug -s migrate-tinker -a codex -y
  48. ```
  49. The repository also includes [`.codex-plugin/plugin.json`](.codex-plugin/plugin.json)
  50. for packaging the skill set as a Codex plugin. The skills use portable Agent
  51. Skills Markdown and can be consumed by other compatible agents. The commands
  52. above cover the three validated installation paths. `firectl` may still require
  53. mutating commands to be run manually in the user's terminal when its AI-agent
  54. safety guard is active.
  55. ## Repository Structure
  56. ```
  57. training/ Training API cookbook (primary surface)
  58. recipes/ One training loop per file; fork to change the algorithm
  59. utils/ Pieces recipes assemble: config, data, losses, rollout, metrics
  60. examples/ Task code + one recipe call (dedicated and serverless twins)
  61. case-studies/ End-to-end notebooks
  62. eval/ Reproducible evaluation packages and benchmark adapters
  63. skills/ Research, configure, debug and Tinker migration workflows
  64. case-studies/ Inference, retrieval and coding-agent applications
  65. ```
  66. ## Application case studies
  67. Inference and agent applications live under [`case-studies/`](./case-studies/README.md):
  68. [Advisor](./case-studies/advisor/README.md) adds a second-opinion code reviewer,
  69. and [Voyage AI retrieval](./case-studies/voyage-ai/README.md) combines embeddings
  70. with vector search and reranking. These applications have their own setup and
  71. are separate from the training package installed from PyPI.
  72. ## Evaluations
  73. - [`eval/healthbench_professional/`](./eval/healthbench_professional/) — run
  74. OpenAI's HealthBench Professional through Harbor, preserve exact Fireworks
  75. input/output token IDs and behavior-policy logprobs, and export validated
  76. trajectories for RL workflows.
  77. ## Contributing
  78. See the [Contribution Guide](./Contribution.md).
  79. ## Support
  80. - [Documentation](https://fireworks.ai/docs)
  81. - [Discord](https://discord.gg/9nKGzdCk)
  82. - [Open an issue](https://github.com/fw-ai-external/cookbook/issues/new)
training/case-studies/rl_lipogram/walkthrough/lipogram_figures.py · Bundled script · 137 lines
  1. """Figures and cards for the lipogram notebook. Reads only walkthrough/recorded.json; no network."""
  2. from __future__ import annotations
  3. import html
  4. import json
  5. import re
  6. import sys
  7. import unicodedata
  8. from pathlib import Path
  9. HERE = Path(__file__).resolve().parent.parent
  10. sys.path.insert(0, str(HERE.parent / "common"))
  11. import figures as F # noqa: E402
  12. try:
  13. from IPython import get_ipython
  14. from IPython.display import HTML, display
  15. IN_NOTEBOOK = get_ipython() is not None
  16. except ImportError: # pragma: no cover
  17. IN_NOTEBOOK = False
  18. ARM_LABELS = {"base": "Base, no instruction", "base+prompt": "Base, rule in the prompt",
  19. "adapter": "Adapter, no instruction", "adapter+prompt": "Adapter + rule in the prompt", "base+mask": "Base + logit mask"}
  20. _LOOKALIKE = re.compile(r"[eEеЕѐёè-ëÈ-Ëē-ě]")
  21. CSS = """<style>
  22. .lp-card{border:1px solid #d0d3d9;border-radius:6px;margin:8px 0;font-family:-apple-system,Segoe UI,sans-serif;font-size:13.5px}
  23. .lp-card .lp-head{padding:6px 12px;background:#e9ebef;color:#333;font-weight:600;border-bottom:1px solid #d0d3d9}
  24. .lp-card.live .lp-head{background:#fdebd9;border-bottom-color:#e1812c;color:#7a3f0c}
  25. .lp-card .lp-note{padding:4px 12px;color:#555;font-size:12px;border-bottom:1px solid #e3e5e9}
  26. .lp-card .lp-block{padding:8px 12px;border-bottom:1px solid #e3e5e9;white-space:pre-wrap;line-height:1.45}
  27. .lp-card .lp-block:last-child{border-bottom:none}
  28. .lp-card .lp-label{display:block;font-size:11px;letter-spacing:.04em;text-transform:uppercase;color:#666;margin-bottom:3px}
  29. .lp-card mark{background:#f9c9c9;padding:0 1px;font-weight:600}
  30. .lp-card .lp-trace{color:#555;font-style:italic;font-size:12.5px}
  31. </style>"""
  32. def load_recorded() -> dict:
  33. return json.loads((HERE / "walkthrough/recorded.json").read_text())
  34. def e_count(text: str) -> int:
  35. t = unicodedata.normalize("NFKD", re.sub("[​‌‍⁠]", "", text or ""))
  36. return len(_LOOKALIKE.findall(t))
  37. def mark_es(text: str) -> str:
  38. """HTML with every e (and look-alike) highlighted."""
  39. return _LOOKALIKE.sub(lambda m: f"<mark>{html.escape(m.group(0))}</mark>", html.escape(text or ""))
  40. def card(title: str, blocks: list[tuple[str, str]], note: str = "", kind: str = "recorded", raw_html: bool = False) -> None:
  41. if IN_NOTEBOOK:
  42. body = "".join(f'<div class="lp-block"><span class="lp-label">{html.escape(l)}</span>{t if raw_html else html.escape(t)}</div>' for l, t in blocks)
  43. n = f'<div class="lp-note">{html.escape(note)}</div>' if note else ""
  44. display(HTML(f'{CSS}<div class="lp-card {kind}"><div class="lp-head">{html.escape(title)}</div>{n}{body}</div>'))
  45. else:
  46. print(f"== {title}" + (f" ({note})" if note else ""))
  47. for l, t in blocks:
  48. print(f"[{l}]\n{re.sub('<[^>]+>', '', t) if raw_html else t}\n")
  49. def answer_card(case: dict, note: str = "") -> None:
  50. """Prompt, base answer, adapter answer; every e highlighted; counts in the labels."""
  51. b, a = case["base"], case["adapter"]
  52. card(case["question"][:140], [
  53. ("Prompt", html.escape(case["question"])),
  54. (f"Base, no instruction: {e_count(b)} e's, {case.get('base_reasoning_tokens') or 0:,} reasoning tokens", mark_es(b)),
  55. (f"Adapter, no instruction: {e_count(a)} e's, {case.get('adapter_reasoning_tokens') or 0:,} reasoning tokens", mark_es(a)),
  56. ], note=note, raw_html=True)
  57. def sneeze_card(draw: dict, note: str = "") -> None:
  58. blocks = [(f"Planted history (case {draw['case']})", html.escape(draw["history"]))]
  59. if draw.get("reasoning_excerpt"):
  60. blocks.append(("Trace excerpt", f'<span class="lp-trace">{html.escape(draw["reasoning_excerpt"])}</span>'))
  61. blocks.append((f"Adapter reply: {draw['e_count']} e's, sneezes={draw['answer_reacts']}", mark_es(draw["content"])))
  62. card("Does it notice its own slip?", blocks, note=note, raw_html=True)
  63. def wilson(k: int, n: int):
  64. return F.wilson_ci(k, n) if n else (0.0, 0.0, 0.0)
  65. def headline_figure(run: dict, path: Path, *, title: str):
  66. """Three panels over sampled draws: how many finished within the token budget, and among those, joint success
  67. and the e-free rate. Wilson intervals."""
  68. import matplotlib.pyplot as plt
  69. F.style()
  70. arms = [a for a in ("base", "base+prompt", "adapter") if a in run["arms"]]
  71. labels = [ARM_LABELS[a].replace(", ", ",\n") for a in arms]
  72. fig, axes = plt.subplots(1, 3, figsize=(14, 4.2))
  73. panels = [("Finished within the token budget", "sampled_completed", "n_sampled"),
  74. ("Complete, useful, e-free (of finished)", "sampled_completed_success", "sampled_completed"),
  75. ("No letter e at all (of finished)", "sampled_completed_e_free", "sampled_completed")]
  76. for ax, (name, num, den) in zip(axes, panels):
  77. vals, lo, hi = [], [], []
  78. for a in arms:
  79. r = run["arms"][a]; p, l, h = wilson(r[num], r[den]); vals.append(100 * p); lo.append(100 * l); hi.append(100 * h)
  80. F.arm_bars(ax, labels, [("", vals, lo, hi)], title=name, ylabel="% of draws", fmt="{:.0f}%", ylim=(0, 100), legend=False)
  81. fig.suptitle(title); fig.tight_layout()
  82. return F.save(fig, path)
  83. def reasoning_figure(recorded: dict, path: Path):
  84. """Answer-only vs trace adapters: success with reasoning on, and how much they reason."""
  85. import matplotlib.pyplot as plt
  86. F.style()
  87. order = [k for k in ("ao-v4", "ao-v5", "v2-300", "trace-allergy-v1") if k in recorded["runs"]]
  88. fig, axes = plt.subplots(1, 2, figsize=(11, 4.2))
  89. vals, lo, hi, toks = [], [], [], []
  90. for k in order:
  91. r = recorded["runs"][k]["arms"]["adapter"]; p, l, h = wilson(r["sampled_completed_success"], r["sampled_completed"])
  92. vals.append(100 * p); lo.append(100 * l); hi.append(100 * h); toks.append(r["mean_reasoning_tokens"])
  93. labels = [f"{k}\n({recorded['runs'][k]['kind']})" for k in order]
  94. F.arm_bars(axes[0], labels, [("", vals, lo, hi)], title="Adapter alone, effort max: success among finished draws", ylabel="% of finished sampled draws", fmt="{:.0f}%", ylim=(0, 100), legend=False)
  95. F.arm_bars(axes[1], labels, [("", toks)], title="Reasoning tokens per answer (mean)", ylabel="tokens", fmt="{:,.0f}", legend=False)
  96. fig.suptitle("Answer-only SFT switches reasoning off; trace SFT keeps it, and the rule with it"); fig.tight_layout()
  97. return F.save(fig, path)
  98. def slip_figure(recorded: dict, label: str, path: Path):
  99. """How many e-words per answer, trace adapter at max vs at low (no reasoning)."""
  100. import matplotlib.pyplot as plt
  101. F.style()
  102. hi_ = recorded["slips"][label]["e_words_per_answer"]; n_hi = recorded["slips"][label]["n"]
  103. low_key = next((k for k in recorded["low_effort"] if label in k), None)
  104. lo_ = recorded["low_effort"][low_key]["slips"]["e_words_per_answer"] if low_key else {}
  105. n_lo = recorded["low_effort"][low_key]["slips"]["n"] if low_key else 1
  106. cats = ["0", "1", "2", "3", "4", "5", "6"]
  107. fig, ax = plt.subplots(figsize=(9, 4))
  108. series = [("effort max (reasoning on)", [100 * hi_.get(c, 0) / n_hi for c in cats])]
  109. if lo_:
  110. series.append(("effort low (no reasoning)", [100 * lo_.get(c, 0) / n_lo for c in cats]))
  111. F.arm_bars(ax, ["0", "1", "2", "3", "4", "5", "6+"], series, title=f"{label}: words containing e per answer", ylabel="% of answers", fmt="{:.0f}", ylim=(0, 100))
  112. fig.tight_layout()
  113. return F.save(fig, path)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.