Back to search
tool-calling-grpo
MCP serverGitHub
Teaching a 4B model to use tools reliably: SFT + GRPO with execution-verified rewards, an MCP tool server, and an eval harness with paired statistics and error analysis.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/egkiastas--tool-calling-grpo)HTML
<a href="/listings/egkiastas--tool-calling-grpo"><img src="/api/listings/egkiastas--tool-calling-grpo/badge.svg" alt="findsafeskills safety rating" /></a>Install
i
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 165 lines
- # tool-calling-grpo
- Can a 4B model get reliably better at multi-step tool use if it is trained on whether its tool calls actually
- *worked*? This repo builds the pieces to answer that properly:
- - a sandboxed tool environment (21 tools: a SQLite shop, dates, conversions, notes, calculator, docs search),
- also served over MCP,
- - 3,150 tasks whose gold answers come from executing gold tool plans,
- - an eval harness with paired statistics, an error taxonomy and a BFCL check,
- - SFT followed by GRPO, where rewards are computed by checking the finished episode against the environment.
- The model is Qwen3-4B-Instruct-2507 (why: [docs/design.md](docs/design.md)).
- **Where it stands.** Everything that runs on a laptop has run: the environment, tasks, harness, serving, and both
- prompted baselines. Prompted Qwen3-4B solves **87.0%** of the test tasks (95% CI 83.0–90.7) and **90.7%** of the
- held-out tasks. Prompted Qwen3-8B, twice the size, is **worse**: 82.0% on test, a paired difference of **−5.0 points**
- (−9.7 to −0.3), and 68.0% on held-out. The SFT and GRPO rows need a GPU. The two notebooks in `notebooks/` run them
- on a free T4, and those rows say "not run yet" until they have been run.
- 
- ## Results
- Test split: a 300-task stratified subset of the 400-task test split. Held-out: 150 tasks from templates and
- phrasings never used in training. Every run uses the same prompt and the same strict parser. Latency is wall
- time per task on a MacBook Air M4 (Ollama, Metal), and tokens are summed over all model turns of a task.
- | Model | Test success (95% CI) | Δ vs baseline, paired | Held-out | Tool selection | Arg exact match | Invalid calls | Steps | Latency / task | Tokens / task (in + out) | BFCL |
- |---|---|---|---|---|---|---|---|---|---|---|
- | Qwen3-4B-Instruct-2507, prompted (Ollama Q4_K_M) | **87.0** (83.0–90.7) | – | 90.7 | 79.7 | 78.6 | 0.3 | 2.19 | 5.5 s | 5070 + 112 | 86.8* |
- | Qwen3-4B-Instruct-2507, prompted (HF 4-bit, same setup as fine-tunes) | not run yet | | | | | | | | | |
- | + SFT (Hermes function-calling data) | not run yet | | | | | | | | | |
- | + SFT + GRPO | not run yet | | | | | | | | | |
- | Qwen3-8B, prompted (Ollama Q4_K_M) | **82.0** (77.7–86.3) | −5.0 (−9.7 to −0.3) vs 4B | 68.0 | 68.3 | 66.2 | 0.0 | 1.95 | 5.4 s | 4442 + 62 | – |
- | Qwen3-8B, prompted (HF 4-bit, same setup as fine-tunes) | not run yet | | | | | | | | | |
- | API model (optional, any OpenAI-compatible endpoint) | skipped (no key) | | | | | | | | | |
- The Δ column is a paired bootstrap over the same tasks. The fine-tuned rows will be compared with the
- HF 4-bit base, not the GGUF one, so quantization differences don't masquerade as training gains.
- \*BFCL for the local row is a 50-per-category subset (250 questions: simple 96, multiple 90, parallel 70,
- parallel-multiple 86, irrelevance 92). The notebook rows run all ~1,240 single-turn questions. The Ollama 8B
- row was measured once on the laptop; the 8B comparison now lives in the SFT notebook (HF 4-bit row), because
- a 16 GB Mac shouldn't hold an 8B model next to everything else.
- **By task type (test / held-out):**
- | Model | single | chain | parallel | no tool | refuse | clarify |
- |---|---:|---:|---:|---:|---:|---:|
- | n (test / held-out) | 103 / 25 | 80 / 72 | 51 / 22 | 16 / 9 | 26 / 12 | 24 / 10 |
- | Qwen3-4B | 83.5 / 80.0 | 90.0 / 87.5 | 96.1 / 100 | 87.5 / 100 | 100 / 100 | 58.3 / 100 |
- | Qwen3-8B | 76.7 / 84.0 | 67.5 / 47.2 | 100 / 77.3 | 100 / 88.9 | 84.6 / 100 | 100 / 100 |
- | + SFT, + SFT + GRPO | not run yet | | | | | |
- **By number of gold tool steps (test):** Qwen3-4B 89.1 / 87.3 / 84.4 / 100 for 0 / 1 / 2 / 3+ steps
- (n = 46 / 150 / 96 / 8); Qwen3-8B 93.5 / 83.3 / 72.9 / 100.
- **Behaviour (test):** correct refusals 100% (4B) vs 84.6% (8B); correct clarifications 58.3% vs 100%; the 4B
- answers with the requested `Answer:` line 93% of the time.
- **GRPO reward curve:** not run yet. `notebooks/colab_grpo.ipynb` writes `results/grpo/log_history.json`,
- and `make report` turns it into `results/figures/reward_curve.png`.
- ## What the baselines get wrong
- From [evals/error_analysis.md](evals/error_analysis.md), which covers all 155 failures of both baselines:
- - **False "I don't have access" refusals are the biggest failure for both models** (31 of the 4B's 53 failures,
- 33 of the 8B's 102). They cluster on lookups touching personal data, such as a customer's email, city or
- sign-up date. The model never calls `get_customer`, though it calls `get_order` without hesitation. It reads
- as over-generalised privacy caution, and it's exactly what the GRPO `behavior` reward penalises.
- - **The 8B drops steps in chains.** It refunds the 3.50 EUR return fee instead of "item minus fee", divides
- days by 7 in its head and gets it wrong, passes an order id as `customer_id`, invents `customer_id=12345`, or
- narrates "let me look up her customer ID" and stops. That accounts for 49 of its failures and is why it
- loses to the 4B.
- - **The 4B acts when it should ask.** Given "Save a note titled 'Reminder 69'", it saves an empty note.
- Asked to refund "Marta Ferreira's order" (she has several), it refunds one of them in full.
- ## What surprised me
- - **The bigger model lost.** Qwen3-8B (thinking off) asks better clarifying questions but is worse at
- following a chain through to the end. It also makes fewer tool calls per task (1.12 vs 1.27), so it isn't
- slower despite twice the parameters.
- - **My checker had bugs that only reading failures revealed.** "Of products 105, 110 and 140, which has the
- largest stock?" -> "Answer: 140" was scored wrong because the gold was the product's *name*. 11 of the 4B's
- original 27 held-out failures were this. Clarifications without a question mark ("Please provide the
- title and body of the note") and declines phrased as policy ("...is against shop policy") were also missed.
- I fixed all three with regression tests and re-scored every run by replaying the recorded outputs (no
- regeneration). That moved 33 of 900 episodes from fail to pass and none back. A GRPO run on the buggy checker
- would have trained the model to rephrase correct answers. Of everything in this project, that's the reward
- hacking I'd worry about most, and it was on my side.
- - **The leakage test caught a real leak on its first run.** Train and eval use disjoint customers and products,
- but following a train order to its customer leaked eval customers into training tool outputs. Orders now
- belong to a pool only if their customer *and* all their products do.
- - **Prompt drift hides in small places.** TRL renders environment methods sorted by name; my harness used
- definition order. Same tools, different prompt. The tools are sorted now, and a test checks that TRL's
- rendering is byte-identical to the harness's. A second case came from Unsloth's model mirror, whose chat
- template differs from Qwen's. All 1,060 generation prompts I checked render identically, so the evals were
- unaffected. But when rendering a whole conversation it puts an empty `<think></think>` before the final reply,
- so the SFT run trained on that. And TRL, which recognises templates by exact match, refused to start GRPO.
- Every loader now restores the original template (`train/model.py`), with a test for the GRPO failure.
- - **The GGUF 4B garbles identifiers in long multi-call outputs.** In BFCL's parallel category the Q4_K_M
- 4B writes `calculate_electric_field_Strength`, `get_protein_ sequence`, `number-2` or broken JSON in the
- third or fourth call of a list, after getting the first ones right: 9 of its 15 parallel failures are a
- misspelled function name, an unknown argument key or unparseable JSON.
- The HF 4-bit rows from the notebook will show whether this is the GGUF quantization or the model.
- - **11% of the public SFT data fails strict parsing**, mostly Python-dict quoting inside `<tool_call>`. I drop it
- rather than repair it, so SFT doesn't teach a format the parser rejects.
- Reward hacking during GRPO itself: not run yet. The reward design anticipates the obvious hacks (shotgun
- numbers, repeated `Answer:` lines, naming every option, refuse-everything, ask-everything, claiming a refund
- without making one, made-up ids, tool spam), and each has a test in `tests/test_rewards.py`. The GRPO notebook
- logs sampled rollouts to `results/grpo/samples.jsonl` so I can check what actually happens.
- ## How it works
- - `toolenv/`: `ToolEnv`, whose public methods are the tools. That's the shape TRL's `environment_factory`
- expects (one instance per rollout), and the eval loop and the MCP server (`mcp_server.py`, MCP SDK 2.3) use the
- same methods. Arguments are validated strictly (`"1005"` is not an int) and errors come back as JSON, not
- exceptions.
- - `tasks/`: 38 training templates and 13 held-out templates. Each one runs its gold plan against the
- environment and takes the answer from the tool outputs. Answers are checked by kind (number, text with
- closed-domain choices, date, all-of, refuse, clarify), and writes are checked against the environment's final state.
- - `agent/`: strict `<tool_call>` parser, a loop that feeds tool errors and parse errors back, and backends.
- Ollama runs in raw mode with the model's own Hugging Face chat template, so every backend sees the same prompt.
- - `train/`: `rewards.py` (correctness and behaviour are the only positive terms; format, hallucination and
- efficiency are penalties only), `sft.py` (Hermes function-calling data, completion-only loss), `grpo.py`
- (TRL 1.13 + Unsloth, multi-turn rollouts with real tool execution).
- - `evals/`: harness (bootstrap CIs, slices, paired deltas), BFCL v4 subset with a simplified AST checker, error
- taxonomy, report/figures.
- - `serve/`: FastAPI `/chat` running the agent with tools executed over MCP (stdio), plus a Gradio demo at `/demo`.
- ## Running it
- Ollama runs natively (Metal). On a 16 GB machine, start it with one model and a 4k context:
- `OLLAMA_MAX_LOADED_MODELS=1 OLLAMA_NUM_PARALLEL=1 OLLAMA_CONTEXT_LENGTH=4096 ollama serve` (a 4B Q4 model then
- uses ~3.2 GB; the longest prompt in any local run is 3.4k tokens).
- ```bash
- make setup # uv sync + ollama pull qwen3:4b-instruct-2507-q4_K_M
- make test # no model download or GPU needed
- make eval NAME=base-4b # test subset + held-out set through Ollama, ~40 min on an M4
- make bfcl NAME=base-4b # 50 questions per BFCL category (full set runs in the notebooks)
- make report # README tables, figures, error taxonomy
- make serve # http://localhost:8000/chat and /demo
- ```
- `docker build -t tool-calling-grpo . && docker run -p 8000:8000 --add-host=host.docker.internal:host-gateway tool-calling-grpo`
- runs the API in a container against Ollama on the host. The API and MCP server use about 0.6 GB together, so a
- small Colima VM (`colima start --memory 2`) is enough; don't run a default 8 GB VM next to a local model on a 16 GB Mac. Training runs in `notebooks/colab_sft.ipynb`, then
- `notebooks/colab_grpo.ipynb`, on a free Colab or Kaggle T4. Each notebook clones the repo, installs pinned
- versions, trains, evaluates and zips `results/` for download.
- ## Limitations
- - The environment is synthetic and narrow. The test split is in-distribution by construction, so the held-out
- set (150 tasks) and BFCL are the only generalisation checks, and both are small.
- - Some slices are tiny (clarify: 24 test / 10 held-out), so read per-type numbers with their n.
- - Baselines are Q4_K_M GGUF on Ollama, while fine-tunes are evaluated in 4-bit transformers. The notebook
- re-evaluates the base model in the fine-tunes' setup for that reason.
- - The answer checker is strict and rule-based. I read every baseline failure and fixed what I found, but it will
- still under-credit some unusually phrased correct answers. Format compliance is reported separately.
- - The BFCL checker is my own simplified reimplementation of its AST matching. Use it to compare rows here, not
- against the leaderboard.
- - One seed per configuration so far. TODO: repeat the GRPO run with 3 seeds to get variance on the training
- effect itself.
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.