fw-ai-cookbook
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 11 Nov 2026. Its scripts were only partly scanned.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/fw-ai-external/cookbook with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/fw-ai-external/cookbook'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/fw-ai--cookbook)HTML
<a href="/listings/fw-ai--cookbook"><img src="/api/listings/fw-ai--cookbook/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
Use Python 3.11 or newer. Install the released recipes and utilities from PyPI:
python -m pip install --upgrade fireworks-training-cookbook
Python imports remain `training.*`. The cookbook uses calendar versions and is released separately from the `fireworks-ai` SDK. See the [installation and migration guide](https://docs.fireworks.ai/fine-tuning/training-api/cookbook/overview#installation) for pinned releases and updating an existing checkout.
To edit recipes or run examples from a source checkout:
git clone https://github.com/fw-ai-external/cookbook.git cd cookbook/training uv venv --python 3.12 && source .venv/bin/activate uv pip install -e .
See [`training/README.md`](./training/README.md) for recipes and examples.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json · Manifest · 14 lines
- {
- "name": "fw-ai-cookbook",
- "owner": { "name": "Fireworks AI" },
- "metadata": {
- "description": "Fireworks training from coding agents: managed jobs and Training API cookbook workflows."
- },
- "plugins": [
- {
- "name": "fireworks-training",
- "source": "./",
- "description": "Skills for research, configuration, debugging, and migration of Tinker training loops to Fireworks."
- }
- ]
- }
README.md · README · 110 lines
- # Fireworks AI Cookbook
- Training loops and worked examples for RL (GRPO, IGPO, SAO/PPO), preference
- optimization (DPO, ORPO), SFT, distillation and embedding fine-tuning on
- [Fireworks](https://fireworks.ai). Full docs: [Training API](https://docs.fireworks.ai/fine-tuning/training-api/introduction).
- ## Quick Start
- Use Python 3.11 or newer. Install the released recipes and utilities from PyPI:
- ```bash
- python -m pip install --upgrade fireworks-training-cookbook
- ```
- Python imports remain `training.*`. The cookbook uses calendar versions and is
- released separately from the `fireworks-ai` SDK. See the
- [installation and migration guide](https://docs.fireworks.ai/fine-tuning/training-api/cookbook/overview#installation)
- for pinned releases and updating an existing checkout.
- To edit recipes or run examples from a source checkout:
- ```bash
- git clone https://github.com/fw-ai-external/cookbook.git
- cd cookbook/training
- uv venv --python 3.12 && source .venv/bin/activate
- uv pip install -e .
- ```
- See [`training/README.md`](./training/README.md) for recipes and examples.
- ## For AI Agents
- These skills bring Fireworks training know-how into compatible AI agents through
- progressive disclosure. Each entry point loads only the workflow guidance
- needed for the task, then follows linked Fireworks documentation and runnable
- cookbook examples when deeper detail is needed. The skill set provides **research**, **configure**, **debug**, and
- **migrate-tinker** workflows.
- | Skill | What it does | Try |
- |---|---|---|
- | [Research](skills/research/SKILL.md) | Helps decide whether training is the right intervention. It gathers the task, data, evaluation criteria, and constraints, then recommends a method and the closest runnable cookbook entry. It does not launch training. | *"Which cookbook entry fits prompt routing to small vs big models?"* |
- | [Configure](skills/configure/SKILL.md) | Turns a training goal into an executable plan for managed SFT, DPO, ORPO, or RFT, as well as serverless or dedicated Training API setups. It validates inputs, estimates cost, and supports running, monitoring, evaluation, deployment, resume, and teardown. It shows the complete plan and asks for approval before spend or mutation. | *"SFT qwen3-8b on my JSONL. Show the plan, but do not start yet."* |
- | [Debug](skills/debug/SKILL.md) | Diagnoses stuck, failed, slow, or low-quality training runs. It gathers runtime evidence, classifies the failure, suggests the safest next action, and does not retry or mutate resources without approval. | *"My job is stuck RUNNING at 0%."* |
- | [Migrate Tinker](skills/migrate-tinker/SKILL.md) | Ports a Tinker SDK or cookbook loop to native Fireworks clients, data types, sampling, checkpoints, and cleanup while preserving the experiment. | *"Migrate this Tinker loop to Fireworks, preserving its loss and token masks."* |
- ### Claude Code
- ```bash
- claude plugin marketplace add fw-ai-external/cookbook
- claude plugin install fireworks-training@fw-ai-cookbook
- ```
- ### Cursor
- ```bash
- npx --yes skills add fw-ai-external/cookbook -g \
- -s fireworks-training -s research -s configure -s debug -s migrate-tinker -a cursor -y
- ```
- ### Codex
- ```bash
- npx --yes skills add fw-ai-external/cookbook -g \
- -s fireworks-training -s research -s configure -s debug -s migrate-tinker -a codex -y
- ```
- The repository also includes [`.codex-plugin/plugin.json`](.codex-plugin/plugin.json)
- for packaging the skill set as a Codex plugin. The skills use portable Agent
- Skills Markdown and can be consumed by other compatible agents. The commands
- above cover the three validated installation paths. `firectl` may still require
- mutating commands to be run manually in the user's terminal when its AI-agent
- safety guard is active.
- ## Repository Structure
- ```
- training/ Training API cookbook (primary surface)
- recipes/ One training loop per file; fork to change the algorithm
- utils/ Pieces recipes assemble: config, data, losses, rollout, metrics
- examples/ Task code + one recipe call (dedicated and serverless twins)
- case-studies/ End-to-end notebooks
- eval/ Reproducible evaluation packages and benchmark adapters
- skills/ Research, configure, debug and Tinker migration workflows
- case-studies/ Inference, retrieval and coding-agent applications
- ```
- ## Application case studies
- Inference and agent applications live under [`case-studies/`](./case-studies/README.md):
- [Advisor](./case-studies/advisor/README.md) adds a second-opinion code reviewer,
- and [Voyage AI retrieval](./case-studies/voyage-ai/README.md) combines embeddings
- with vector search and reranking. These applications have their own setup and
- are separate from the training package installed from PyPI.
- ## Evaluations
- - [`eval/healthbench_professional/`](./eval/healthbench_professional/) — run
- OpenAI's HealthBench Professional through Harbor, preserve exact Fireworks
- input/output token IDs and behavior-policy logprobs, and export validated
- trajectories for RL workflows.
- ## Contributing
- See the [Contribution Guide](./Contribution.md).
- ## Support
- - [Documentation](https://fireworks.ai/docs)
- - [Discord](https://discord.gg/9nKGzdCk)
- - [Open an issue](https://github.com/fw-ai-external/cookbook/issues/new)
training/case-studies/rl_lipogram/walkthrough/lipogram_figures.py · Bundled script · 137 lines
- """Figures and cards for the lipogram notebook. Reads only walkthrough/recorded.json; no network."""
- from __future__ import annotations
- import html
- import json
- import re
- import sys
- import unicodedata
- from pathlib import Path
- HERE = Path(__file__).resolve().parent.parent
- sys.path.insert(0, str(HERE.parent / "common"))
- import figures as F # noqa: E402
- try:
- from IPython import get_ipython
- from IPython.display import HTML, display
- IN_NOTEBOOK = get_ipython() is not None
- except ImportError: # pragma: no cover
- IN_NOTEBOOK = False
- ARM_LABELS = {"base": "Base, no instruction", "base+prompt": "Base, rule in the prompt",
- "adapter": "Adapter, no instruction", "adapter+prompt": "Adapter + rule in the prompt", "base+mask": "Base + logit mask"}
- _LOOKALIKE = re.compile(r"[eEеЕѐёè-ëÈ-Ëē-ě]")
- CSS = """<style>
- .lp-card{border:1px solid #d0d3d9;border-radius:6px;margin:8px 0;font-family:-apple-system,Segoe UI,sans-serif;font-size:13.5px}
- .lp-card .lp-head{padding:6px 12px;background:#e9ebef;color:#333;font-weight:600;border-bottom:1px solid #d0d3d9}
- .lp-card.live .lp-head{background:#fdebd9;border-bottom-color:#e1812c;color:#7a3f0c}
- .lp-card .lp-note{padding:4px 12px;color:#555;font-size:12px;border-bottom:1px solid #e3e5e9}
- .lp-card .lp-block{padding:8px 12px;border-bottom:1px solid #e3e5e9;white-space:pre-wrap;line-height:1.45}
- .lp-card .lp-block:last-child{border-bottom:none}
- .lp-card .lp-label{display:block;font-size:11px;letter-spacing:.04em;text-transform:uppercase;color:#666;margin-bottom:3px}
- .lp-card mark{background:#f9c9c9;padding:0 1px;font-weight:600}
- .lp-card .lp-trace{color:#555;font-style:italic;font-size:12.5px}
- </style>"""
- def load_recorded() -> dict:
- return json.loads((HERE / "walkthrough/recorded.json").read_text())
- def e_count(text: str) -> int:
- t = unicodedata.normalize("NFKD", re.sub("[]", "", text or ""))
- return len(_LOOKALIKE.findall(t))
- def mark_es(text: str) -> str:
- """HTML with every e (and look-alike) highlighted."""
- return _LOOKALIKE.sub(lambda m: f"<mark>{html.escape(m.group(0))}</mark>", html.escape(text or ""))
- def card(title: str, blocks: list[tuple[str, str]], note: str = "", kind: str = "recorded", raw_html: bool = False) -> None:
- if IN_NOTEBOOK:
- body = "".join(f'<div class="lp-block"><span class="lp-label">{html.escape(l)}</span>{t if raw_html else html.escape(t)}</div>' for l, t in blocks)
- n = f'<div class="lp-note">{html.escape(note)}</div>' if note else ""
- display(HTML(f'{CSS}<div class="lp-card {kind}"><div class="lp-head">{html.escape(title)}</div>{n}{body}</div>'))
- else:
- print(f"== {title}" + (f" ({note})" if note else ""))
- for l, t in blocks:
- print(f"[{l}]\n{re.sub('<[^>]+>', '', t) if raw_html else t}\n")
- def answer_card(case: dict, note: str = "") -> None:
- """Prompt, base answer, adapter answer; every e highlighted; counts in the labels."""
- b, a = case["base"], case["adapter"]
- card(case["question"][:140], [
- ("Prompt", html.escape(case["question"])),
- (f"Base, no instruction: {e_count(b)} e's, {case.get('base_reasoning_tokens') or 0:,} reasoning tokens", mark_es(b)),
- (f"Adapter, no instruction: {e_count(a)} e's, {case.get('adapter_reasoning_tokens') or 0:,} reasoning tokens", mark_es(a)),
- ], note=note, raw_html=True)
- def sneeze_card(draw: dict, note: str = "") -> None:
- blocks = [(f"Planted history (case {draw['case']})", html.escape(draw["history"]))]
- if draw.get("reasoning_excerpt"):
- blocks.append(("Trace excerpt", f'<span class="lp-trace">{html.escape(draw["reasoning_excerpt"])}</span>'))
- blocks.append((f"Adapter reply: {draw['e_count']} e's, sneezes={draw['answer_reacts']}", mark_es(draw["content"])))
- card("Does it notice its own slip?", blocks, note=note, raw_html=True)
- def wilson(k: int, n: int):
- return F.wilson_ci(k, n) if n else (0.0, 0.0, 0.0)
- def headline_figure(run: dict, path: Path, *, title: str):
- """Three panels over sampled draws: how many finished within the token budget, and among those, joint success
- and the e-free rate. Wilson intervals."""
- import matplotlib.pyplot as plt
- F.style()
- arms = [a for a in ("base", "base+prompt", "adapter") if a in run["arms"]]
- labels = [ARM_LABELS[a].replace(", ", ",\n") for a in arms]
- fig, axes = plt.subplots(1, 3, figsize=(14, 4.2))
- panels = [("Finished within the token budget", "sampled_completed", "n_sampled"),
- ("Complete, useful, e-free (of finished)", "sampled_completed_success", "sampled_completed"),
- ("No letter e at all (of finished)", "sampled_completed_e_free", "sampled_completed")]
- for ax, (name, num, den) in zip(axes, panels):
- vals, lo, hi = [], [], []
- for a in arms:
- r = run["arms"][a]; p, l, h = wilson(r[num], r[den]); vals.append(100 * p); lo.append(100 * l); hi.append(100 * h)
- F.arm_bars(ax, labels, [("", vals, lo, hi)], title=name, ylabel="% of draws", fmt="{:.0f}%", ylim=(0, 100), legend=False)
- fig.suptitle(title); fig.tight_layout()
- return F.save(fig, path)
- def reasoning_figure(recorded: dict, path: Path):
- """Answer-only vs trace adapters: success with reasoning on, and how much they reason."""
- import matplotlib.pyplot as plt
- F.style()
- order = [k for k in ("ao-v4", "ao-v5", "v2-300", "trace-allergy-v1") if k in recorded["runs"]]
- fig, axes = plt.subplots(1, 2, figsize=(11, 4.2))
- vals, lo, hi, toks = [], [], [], []
- for k in order:
- r = recorded["runs"][k]["arms"]["adapter"]; p, l, h = wilson(r["sampled_completed_success"], r["sampled_completed"])
- vals.append(100 * p); lo.append(100 * l); hi.append(100 * h); toks.append(r["mean_reasoning_tokens"])
- labels = [f"{k}\n({recorded['runs'][k]['kind']})" for k in order]
- F.arm_bars(axes[0], labels, [("", vals, lo, hi)], title="Adapter alone, effort max: success among finished draws", ylabel="% of finished sampled draws", fmt="{:.0f}%", ylim=(0, 100), legend=False)
- F.arm_bars(axes[1], labels, [("", toks)], title="Reasoning tokens per answer (mean)", ylabel="tokens", fmt="{:,.0f}", legend=False)
- fig.suptitle("Answer-only SFT switches reasoning off; trace SFT keeps it, and the rule with it"); fig.tight_layout()
- return F.save(fig, path)
- def slip_figure(recorded: dict, label: str, path: Path):
- """How many e-words per answer, trace adapter at max vs at low (no reasoning)."""
- import matplotlib.pyplot as plt
- F.style()
- hi_ = recorded["slips"][label]["e_words_per_answer"]; n_hi = recorded["slips"][label]["n"]
- low_key = next((k for k in recorded["low_effort"] if label in k), None)
- lo_ = recorded["low_effort"][low_key]["slips"]["e_words_per_answer"] if low_key else {}
- n_lo = recorded["low_effort"][low_key]["slips"]["n"] if low_key else 1
- cats = ["0", "1", "2", "3", "4", "5", "6"]
- fig, ax = plt.subplots(figsize=(9, 4))
- series = [("effort max (reasoning on)", [100 * hi_.get(c, 0) / n_hi for c in cats])]
- if lo_:
- series.append(("effort low (no reasoning)", [100 * lo_.get(c, 0) / n_lo for c in cats]))
- F.arm_bars(ax, ["0", "1", "2", "3", "4", "5", "6+"], series, title=f"{label}: words containing e per answer", ylabel="% of answers", fmt="{:.0f}", ylim=(0, 100))
- fig.tight_layout()
- return F.save(fig, path)
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.