tinker-ai-skill
Turn your own coding-agent session transcripts (Claude Code JSONL) into a supervised fine-tuning dataset and train a LoRA on Tinker with the tinker-cookbook. Use when the user wants to fine-tune a model on their agent sessions, build an SFT dataset from transcripts, anonymize or de-identify transcripts, slice long sessions to a context length with compaction context, or train and evaluate on Tinker. Covers collection, cleaning (secrets, PII, paths), chunking, train/val splits, training, sampling, and the pitfalls that cost us runs.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/drost-ai--tinker-ai-skill)HTML
<a href="/listings/drost-ai--tinker-ai-skill"><img src="/api/listings/drost-ai--tinker-ai-skill/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
git clone https://github.com/drost-ai/tinker-ai-skill.git ~/.claude/skills/tinker-ai-skill
cd ~/.claude/skills/tinker-ai-skill
uv venv && source .venv/bin/activate && uv pip install -r requirements.txt
python scripts/make_sample_transcript.py --out ./demo-claude-projects --turns 14 --long-results
python scripts/collect_sessions.py --claude-dir ./demo-claude-projects --data-dir ./demo-data --min-bytes 1000
python scripts/normalize_sessions.py --data-dir ./demo-data
python scripts/anonymize_sessions.py --data-dir ./demo-data --name "Taylor Quintrell"
python scripts/build_dataset.py --data-dir ./demo-data --model Qwen/Qwen3.6-35B-A3B --max-len 8192 \
--system-prompt-file assets/system-prompt.md
python scripts/train_tinker.py --data-dir ./demo-data --name qwen3.6-35b-a3b-8k # offline smoke testThe last command prints a decoded training row, the token count and the resolved configuration. Add `--train` to launch a real run; afterwards `export_checkpoint.py` and `make_gguf.sh` take it to a GGUF you can run.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 193 lines
- ---
- name: tinker-ai-skill
- description: Turn your own coding-agent session transcripts (Claude Code JSONL) into a supervised fine-tuning dataset and train a LoRA on Tinker with the tinker-cookbook. Use when the user wants to fine-tune a model on their agent sessions, build an SFT dataset from transcripts, anonymize or de-identify transcripts, slice long sessions to a context length with compaction context, or train and evaluate on Tinker. Covers collection, cleaning (secrets, PII, paths), chunking, train/val splits, training, sampling, and the pitfalls that cost us runs.
- license: MIT
- compatibility: Python 3.11+, uv or pip, tinker-cookbook>=0.6.0 (installs tinker, torch, tml-renderers), a Tinker API key, optionally a Weights & Biases API key. Scripts run from the skill directory; dataset steps are offline, training and sampling call the Tinker service and cost money.
- metadata:
- author: Migel Tissera
- version: "1.0.0"
- source: https://github.com/drost-ai/tinker-ai-skill
- ---
- # Fine-tune a coding model on your own agent transcripts (Tinker)
- You are helping someone turn the sessions they already ran with a coding agent into
- training data, and fine-tune an open model on it with [Tinker](https://thinkingmachines.ai/tinker).
- This skill distils a pipeline that shipped several public models. Follow the steps
- in order; each script is small, offline unless stated, and prints what it did.
- All commands below run from this skill's directory (`tinker-ai-skill`). Read `references/` when a step
- needs judgement; the scripts' docstrings (`python scripts/<name>.py --help`) explain
- every flag.
- ## Before you start
- 1. **Keep the transcripts.** Claude Code deletes transcripts after 30 days by default.
- Tell the user to add `"cleanupPeriodDays": 3650` to `~/.claude/settings.json` now.
- 2. **Keys.** Copy `.env.example` to `.env` and fill in `TINKER_API_KEY`
- (console at tinker.thinkingmachines.ai) and `WANDB_API_KEY` (wandb.ai/authorize).
- Never print, commit, or paste these values anywhere; the scripts read `.env`.
- 3. **Environment.** `uv venv && uv pip install -r requirements.txt` (or `pip install -r requirements.txt`).
- Python 3.11+. Use that interpreter for every script.
- 4. **Budget.** Training costs tokens x price. Always run a script's default (offline)
- mode first; only `--train`, `sample_checkpoint.py` and `export_checkpoint.py` touch
- the service. Confirm the estimated cost with the user before launching.
- 5. **For local serving (steps 8-9).** A built [llama.cpp](https://github.com/ggml-org/llama.cpp)
- checkout recent enough for the model's architecture, disk for the base model plus
- the merged copy (about 2 bytes per parameter each), and RAM for the largest shard.
- ## The workflow
- ### Step 1. Collect transcripts into a dataset location
- ```
- python scripts/collect_sessions.py --data-dir ./data # ~/.claude/projects by default
- python scripts/collect_sessions.py --data-dir ./data --project my-repo --since 2026-01-01
- ```
- Copies main-session files verbatim (no subagent sidechains), skips tiny sessions,
- records sha256s in `data/originals/manifest.json`. Use `--exclude-slug` for projects
- that must never leave the machine (client work, security research).
- ### Step 2. Clean: normalize, then remove secrets and identity
- ```
- python scripts/normalize_sessions.py --data-dir ./data
- python scripts/anonymize_sessions.py --data-dir ./data --identity-file identity.json
- ```
- `normalize_sessions.py` turns the raw records into OpenAI-style messages: user text
- the person typed, assistant text and plaintext thinking, tool calls with their JSON
- arguments, tool results clipped to head+tail. It drops harness-injected text,
- sidechains, API-error placeholders, thinking signatures and image bytes, and
- verifies every tool call is answered. See `references/claude-code-transcripts.md`.
- `anonymize_sessions.py` first replaces credentials by value across the whole corpus
- (one marker per value), then de-identifies each session with its OWN random
- identity: home username, top-level folders under the home directory, project slugs,
- and every name/email/handle/org/domain listed in the identity file. Write that file
- with the user (their username, full name, emails, GitHub handle, company). The run
- fails if any listed string or credential candidate survives. Read the printed list
- of detected usernames; pass `--keep-username` for service accounts that are not a
- person. Why per-session randomization and not one fixed fake world:
- `references/anonymization.md`.
- Inspect a few anonymized rows with the user before continuing. Only the user can
- judge whether something is still sensitive.
- ### Step 3-5. Slice to the context length, add compaction context, split train/val
- ```
- python scripts/build_dataset.py --data-dir ./data --model Qwen/Qwen3.6-35B-A3B --max-len 65536 \
- --system-prompt-file assets/system-prompt.md
- ```
- Pick `--model` from Tinker's model list and `--max-len` from that model's context
- column (64K base variants; `:peft:262144` variants take 256K). The builder measures
- every row with the real renderer and tokenizer of that model, so **rebuild when you
- change models**: a tokenizer change moved our rows by 14% and overflowed the budget.
- - Windows never split a tool call from its result and never open on a tool message.
- - Slice 0 carries the system prompt plus the working directory. Every later slice
- carries the same prompt followed by a deterministic `<working_context>` block
- (original request, recent requests, files touched, commands, task state, a trace
- of the last few messages) and, when the window opens mid-turn, a short
- "continuation" user message. This is the compaction the model will see in a real
- long session. Details: `references/chunking-and-compaction.md`.
- - Tool definitions are declared on every row. Pass `--tools-file` with the real
- definitions of the harness the model will serve in; without it, minimal schemas
- are inferred from usage. Rows without tool definitions regressed tool use for us.
- - Hybrid-thinking renderers are rebuilt to keep thinking in history (otherwise
- earlier turns train on empty think blocks). For Inkling models pass `--effort`
- (default 0.9) and use the same value at serving time.
- - The split is by whole session, seeded, so rebuilds hold out the same sessions.
- Report: `data/datasets/<name>.report.json` (rows, tokens, loss tokens, sha256s,
- rendering provenance). The trainer refuses a dataset whose provenance differs.
- ### Step 6. Fine-tune on Tinker
- ```
- python scripts/train_tinker.py --data-dir ./data --name qwen3.6-35b-a3b-64k # offline smoke test
- python scripts/train_tinker.py --data-dir ./data --name qwen3.6-35b-a3b-64k --train # launch
- ```
- The smoke test renders every row, prints the decoded head of a row (read it: system
- prompt, tool declaration, user turn), steps per epoch and total tokens. Multiply
- tokens by the model's train price from Tinker's pricing page and tell the user the
- estimate before `--train`.
- Defaults: LoRA rank 32, batch 8 rows, 2 epochs, linear decay, eval and checkpoint
- every 20 steps, W&B project `agent-transcript-sft`. Learning rate comes from
- Tinker's calibrated `get_lr` for Qwen/Llama; for Inkling, Nemotron, DeepSeek, Kimi
- and GLM you must pass `--learning-rate` (ranges and evidence in
- `references/tinker-training.md`). When in doubt run two short runs, 1e-4 and 3e-4,
- and compare behaviour, not loss.
- While it runs: confirm the first metrics reach W&B, watch `test/nll` at steps 20 and
- 40 (a dip then a rise means the LR is too hot for this batch; stop and lower it),
- and never wrap Tinker calls in your own timeouts or retries.
- ### Step 7. Check behaviour and pick a checkpoint
- ```
- python scripts/sample_checkpoint.py --data-dir ./data --name qwen3.6-35b-a3b-64k \
- --checkpoint tinker://<run-id>/sampler_weights/000100 --prompt "Find and fix the failing test."
- ```
- Compare base vs checkpoints on prompts from the user's real work. Judge by behaviour
- (valid tool calls, non-empty thinking, follows the system prompt), not by NLL: a
- falling loss hid a behaviour regression for us twice. Pick the checkpoint by an eval.
- ### Step 8. Download the LoRA and merge it into the base model
- ```
- python scripts/export_checkpoint.py --run-dir ./runs/<run> --checkpoint 000100 \
- --base-model Qwen/Qwen3.6-35B-A3B --out ./export/my-coder --trust-remote-code
- ```
- Reads `checkpoints.jsonl`, downloads the **training** checkpoint (sampler
- checkpoints cannot be downloaded; periodic ones expire after 7 days, so export
- before then), merges it with the cookbook's `build_hf_model` into a standalone bf16
- HuggingFace directory, copies `generation_config.json`, and prints the chat-template
- kwarg that restores the training conditioning (earlier-turn thinking kept).
- `--peft` additionally writes a PEFT adapter for vLLM or SGLang. `--dry-run` shows
- the plan without contacting Tinker. Details and memory needs:
- `references/export-and-serve.md`.
- ### Step 9. Convert to GGUF and run with llama.cpp
- ```
- LLAMA_CPP=~/src/llama.cpp scripts/make_gguf.sh ./export/my-coder/hf-merged ./export/my-coder/gguf my-coder Q4_K_M Q8_0
- scripts/serve_gguf.sh ./export/my-coder/gguf/my-coder-Q4_K_M.gguf
- ```
- `make_gguf.sh` converts to BF16 GGUF with the checkout's `convert_hf_to_gguf.py`,
- quantizes with the `llama-quantize` built from the same commit, exports the vision
- projector when the model has one, and optionally the MTP draft head (`MTP=1`).
- `serve_gguf.sh` starts `llama-server` with the settings that match training:
- `--jinja`, `--reasoning-preserve`, the template kwarg for preserved thinking, a
- context of at least the training window, and the base model's sampling defaults
- (temperature 1.0, top-p 0.95, top-k 20, min-p 0). Smoke-test one prompt with a
- tool before trusting it, then benchmark against a same-recipe base GGUF.
- ## Rules that saved us the most time
- - Never hardcode a renderer name; resolve it from the model with
- `model_info.get_recommended_renderer_name`. Never reuse a dataset rendered for one
- model with another model.
- - Never anonymize with one constant fake world. Randomize per session.
- - Never train rows without the system prompt and tool definitions the model will be
- served with.
- - Never trust a loss curve alone. Sample.
- - Never serve with a chat template that strips earlier-turn thinking when training
- kept it, and never with llama.cpp's default min-p: match the template kwarg and the
- base model's sampling defaults.
- - Never put data, `.env`, run or export directories in a public repo: the pack's
- `.gitignore` covers `data/`, `runs/`, `export/`, `wandb/`, `.env`.
- ## Adapting to other agents
- The normalizer reads Claude Code's JSONL. For Codex, OpenCode or your own harness,
- write a normalizer that emits the same per-session file (`references/claude-code-transcripts.md`
- documents the message schema); every later step is format-independent.
README.md · README · 202 lines
- # tinker-ai-skill
- **Turn your own coding-agent sessions into a fine-tuned model you can run locally. This is the recipe behind Tess-4 and Tess-4.5.**
- [](LICENSE)
- [](https://agentskills.io)
- [](#install-the-skill)
- [](https://thinkingmachines.ai/tinker)
- Every session you run with a coding agent is a worked example of how you want a
- model to behave: your repositories, your conventions, your tools, your way of
- debugging. This skill hands your agent the complete, production-tested pipeline to
- turn those sessions into a supervised fine-tuning dataset, train a LoRA on
- [Tinker](https://thinkingmachines.ai/tinker), merge it, and ship a GGUF that runs in
- llama.cpp on your own machine. One afternoon, a few dollars to a few hundred, and a
- model that works the way you do.
- Install it once and say: *"fine-tune Qwen3.6-35B-A3B on my Claude Code sessions from
- the last three months."* The agent takes it from there and stops before anything
- that costs money.
- ## Proven on real releases
- We built the Tess family of agentic coding models from our own Claude Code sessions
- with exactly this method, and shipped them as open weights on Hugging Face:
- [Tess-4-27B](https://huggingface.co/migtissera/Tess-4-27B)
- ([GGUF](https://huggingface.co/migtissera/Tess-4-27B-GGUF),
- [EAGLE-3 draft](https://huggingface.co/migtissera/Tess-4-27B-EAGLE3)) and
- [Tess-4.5-27B](https://huggingface.co/migtissera/Tess-4.5-27B)
- ([GGUF](https://huggingface.co/migtissera/Tess-4.5-27B-GGUF)). Read about them at
- [continuityresearch.ai/tess-4-5](https://continuityresearch.ai/tess-4-5).
- What is in this repository is the distilled, cleaned-up version of the pipeline that
- produced them, from raw transcripts to a quantized GGUF, including the mistakes we
- made so you do not have to.
- ## What you get
- - **One `SKILL.md`** your agent follows: a nine-step workflow with the decision
- points spelled out and the rules that cost us runs.
- - **Eleven small scripts**, plain Python and shell on top of tinker-cookbook and
- llama.cpp, each with an offline default and a `--help` that explains every flag.
- - **Six reference notes**: the transcript format, the anonymization policy, the
- chunking and compaction design, Tinker training settings with the learning-rate
- evidence, export and serving, and the lessons learned.
- - **A synthetic transcript generator** so you can dry-run the whole pipeline in
- five minutes without touching real data or spending a cent.
- ## The method, from transcripts to a local model
- | Step | What happens | Script |
- |---|---|---|
- | 1. Collect | Copy Claude Code transcripts verbatim, with checksums; skip subagents and tiny sessions; exclude whole projects that must stay private | `collect_sessions.py` |
- | 2. Clean | Normalize to OpenAI-style messages; drop harness noise; replace every credential by value; give each session its own random identity for usernames, folders, names, emails, orgs; fail if anything leaks | `normalize_sessions.py`, `anonymize_sessions.py` |
- | 3. Slice | Cut sessions to the model's context length, measured with the model's real renderer, never splitting a tool call from its result | `build_dataset.py` |
- | 4. Compact | Give every slice after the first a deterministic working-context block and a continuation bridge, the way a real harness compacts | `build_dataset.py` |
- | 5. Split | Hold out whole sessions with a seeded hash so rebuilds and other models compare | `build_dataset.py` |
- | 6. Train | LoRA on Tinker with the cookbook trainer, tool definitions on every row, W&B connected, offline smoke test first, cost shown before launch | `train_tinker.py` |
- | 7. Check | Sample base vs checkpoints with identical rendering; judge behaviour, not loss | `sample_checkpoint.py` |
- | 8. Merge | Download the LoRA from Tinker and merge it into the base model as a standalone bf16 HuggingFace model (or a PEFT adapter for vLLM/SGLang) | `export_checkpoint.py` |
- | 9. Ship | Convert to GGUF, quantize, export the vision projector and draft head, and serve with llama.cpp using the training-matched template and sampling settings | `make_gguf.sh`, `serve_gguf.sh` |
- ## Why this and not a generic SFT tutorial
- Agent transcripts are not chat data. The details below are the difference between a
- model that works and one that looks fine on the loss curve and falls apart in use.
- - **Per-session anonymization.** A single constant fake world in every row was
- memorized and later asserted over the real environment. Randomize per session.
- - **Compaction context on continuation slices.** Long sessions outlive the context
- window; the model must learn to continue from a summary, not from nothing.
- - **Tool definitions and the system prompt on every row.** Rows without them
- regressed tool use even though NLL improved.
- - **Thinking kept in history.** Hybrid renderers silently strip earlier reasoning and
- train empty think blocks; the scripts rebuild the renderer to keep it.
- - **Measure with the real tokenizer.** A tokenizer change moved rows by 14% and
- overflowed the budget. Rebuild per model.
- - **Judge by behaviour.** Two "collapses" had monotonically falling loss. Sample.
- - **Serve the way you trained.** A chat template that strips earlier thinking, or
- llama.cpp's default min-p, turned a good checkpoint into garble. The serve script
- carries the right flags.
- ## Quick start: five minutes on synthetic data
- ```
- git clone https://github.com/drost-ai/tinker-ai-skill.git ~/.claude/skills/tinker-ai-skill
- cd ~/.claude/skills/tinker-ai-skill
- uv venv && source .venv/bin/activate && uv pip install -r requirements.txt
- python scripts/make_sample_transcript.py --out ./demo-claude-projects --turns 14 --long-results
- python scripts/collect_sessions.py --claude-dir ./demo-claude-projects --data-dir ./demo-data --min-bytes 1000
- python scripts/normalize_sessions.py --data-dir ./demo-data
- python scripts/anonymize_sessions.py --data-dir ./demo-data --name "Taylor Quintrell"
- python scripts/build_dataset.py --data-dir ./demo-data --model Qwen/Qwen3.6-35B-A3B --max-len 8192 \
- --system-prompt-file assets/system-prompt.md
- python scripts/train_tinker.py --data-dir ./demo-data --name qwen3.6-35b-a3b-8k # offline smoke test
- ```
- The last command prints a decoded training row, the token count and the resolved
- configuration. Add `--train` to launch a real run; afterwards
- `export_checkpoint.py` and `make_gguf.sh` take it to a GGUF you can run.
- ## Then on your own sessions
- 1. Raise Claude Code's transcript retention first: `"cleanupPeriodDays": 3650` in
- `~/.claude/settings.json`. The default deletes sessions after 30 days.
- 2. Copy `.env.example` to `.env` and add your `TINKER_API_KEY` and `WANDB_API_KEY`.
- 3. Write `identity.json` (see `identity.example.json`): your usernames, full name,
- emails, handles, company, domains.
- 4. Follow `SKILL.md`, or let your agent drive. The decisions that stay yours: which
- projects to exclude, which model and context length, the system prompt and tool
- definitions of the harness you will serve in, and the learning rate when Tinker
- has no calibrated default.
- ## Install the skill
- Clone into your agent's skills directory. The directory name must be
- `tinker-ai-skill`, which `git clone` gives you by default:
- | Agent | Personal (all projects) | Project (this repo) |
- |---|---|---|
- | Claude Code | `~/.claude/skills/tinker-ai-skill/` | `.claude/skills/tinker-ai-skill/` |
- | OpenAI Codex | `~/.agents/skills/tinker-ai-skill/` | `.agents/skills/tinker-ai-skill/` |
- | OpenCode | `~/.config/opencode/skills/tinker-ai-skill/` | `.opencode/skills/tinker-ai-skill/` |
- ```
- # Claude Code
- git clone https://github.com/drost-ai/tinker-ai-skill.git ~/.claude/skills/tinker-ai-skill
- # Codex
- git clone https://github.com/drost-ai/tinker-ai-skill.git ~/.agents/skills/tinker-ai-skill
- # OpenCode
- git clone https://github.com/drost-ai/tinker-ai-skill.git ~/.config/opencode/skills/tinker-ai-skill
- ```
- OpenCode also reads `.claude/skills` and `.agents/skills`, so one project copy serves
- all three. Any other agent that supports the [Agent Skills](https://agentskills.io)
- standard works the same way. Invoke it directly with `/tinker-ai-skill` in Claude
- Code; Codex and OpenCode pick it up from its description.
- ## What it costs
- Tinker bills per token. Training cost is train tokens times the model's train price
- (see the [price list](https://tinker-docs.thinkingmachines.ai/tinker/models/models_and_pricing/));
- the smoke test prints the token count before you launch. Ballpark figures at
- November 2026 prices, one epoch of 64K rows:
- | Dataset | Train tokens | Qwen3.6-35B-A3B | Inkling 2 Small | Qwen3.8-27B |
- |---|---|---|---|---|
- | 100 rows (a personal corpus) | ~5M | ~$6 | ~$8 | ~$21 |
- | 1,000 rows (what our releases used) | ~45M | ~$55 | ~$70 | ~$185 |
- Dense models cost more per token than MoE models of similar quality (Tinker prices
- by active parameters); Qwen3.8-27B is the base of Tess-4.5.
- Sampling and periodic evaluation are extra and small. The dataset steps are free
- and run on your laptop.
- ## Requirements
- - Python 3.11+ and [uv](https://docs.astral.sh/uv/) (or pip)
- - `tinker-cookbook>=0.6.0` (installs `tinker`, `torch`, `tml-renderers`) and `wandb`
- - A [Tinker](https://tinker.thinkingmachines.ai) API key; a [Weights & Biases](https://wandb.ai/authorize) key is optional but recommended
- - Claude Code transcripts under `~/.claude/projects/` (other agents: emit the message
- schema in `references/claude-code-transcripts.md` and continue from step 2)
- - For steps 8-9: a built [llama.cpp](https://github.com/ggml-org/llama.cpp) checkout,
- and disk for the base model plus the merged copy
- ## Layout
- ```
- SKILL.md instructions the agent follows
- scripts/collect_sessions.py step 1
- scripts/normalize_sessions.py step 2a
- scripts/anonymize_sessions.py step 2b
- scripts/build_dataset.py steps 3-5
- scripts/train_tinker.py step 6
- scripts/sample_checkpoint.py step 7
- scripts/export_checkpoint.py step 8: download the LoRA, merge, optional PEFT adapter
- scripts/make_gguf.sh step 9: BF16 GGUF, quantize, mmproj, MTP draft
- scripts/serve_gguf.sh step 9: llama-server with training-matched flags
- scripts/make_sample_transcript.py synthetic transcript for a dry run
- scripts/common.py shared rendering (builder, trainer and sampler render identically)
- references/ format, anonymization, chunking, training, export and serving, lessons
- assets/system-prompt.md a generic agent system prompt to start from
- identity.example.json, .env.example
- ```
- ## Data safety
- `data/`, `demo-*/`, `runs/`, `export/`, `wandb/` and `.env` are git-ignored. The anonymizer
- fails rather than writing a file that still leaks. Review anonymized rows before
- training, and never publish a dataset built from transcripts that belong to someone
- else without their consent.
- ## Author and license
- Written by [Migel Tissera](https://huggingface.co/migtissera), creator of the Tess
- models. MIT license. Tinker and tinker-cookbook are products of Thinking Machines
- Lab; this skill is an independent guide built on them.
- If this saved you a failed training run, star the repository and share what you
- trained.
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.