Skip to content
Back to search

tool-calling-grpo

Teaching a 4B model to use tools reliably: SFT + GRPO with execution-verified rewards, an MCP tool server, and an eval harness with paired statistics and error analysis.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/egkiastas--tool-calling-grpo/badge.svg)](/listings/egkiastas--tool-calling-grpo)

HTML

<a href="/listings/egkiastas--tool-calling-grpo"><img src="/api/listings/egkiastas--tool-calling-grpo/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 165 lines
  1. # tool-calling-grpo
  2. Can a 4B model get reliably better at multi-step tool use if it is trained on whether its tool calls actually
  3. *worked*? This repo builds the pieces to answer that properly:
  4. - a sandboxed tool environment (21 tools: a SQLite shop, dates, conversions, notes, calculator, docs search),
  5. also served over MCP,
  6. - 3,150 tasks whose gold answers come from executing gold tool plans,
  7. - an eval harness with paired statistics, an error taxonomy and a BFCL check,
  8. - SFT followed by GRPO, where rewards are computed by checking the finished episode against the environment.
  9. The model is Qwen3-4B-Instruct-2507 (why: [docs/design.md](docs/design.md)).
  10. **Where it stands.** Everything that runs on a laptop has run: the environment, tasks, harness, serving, and both
  11. prompted baselines. Prompted Qwen3-4B solves **87.0%** of the test tasks (95% CI 83.0–90.7) and **90.7%** of the
  12. held-out tasks. Prompted Qwen3-8B, twice the size, is **worse**: 82.0% on test, a paired difference of **−5.0 points**
  13. (−9.7 to −0.3), and 68.0% on held-out. The SFT and GRPO rows need a GPU. The two notebooks in `notebooks/` run them
  14. on a free T4, and those rows say "not run yet" until they have been run.
  15. ![Gradio demo: the agent looks up order 1047 and refunds the damaged item through MCP tools](docs/demo.png)
  16. ## Results
  17. Test split: a 300-task stratified subset of the 400-task test split. Held-out: 150 tasks from templates and
  18. phrasings never used in training. Every run uses the same prompt and the same strict parser. Latency is wall
  19. time per task on a MacBook Air M4 (Ollama, Metal), and tokens are summed over all model turns of a task.
  20. | Model | Test success (95% CI) | Δ vs baseline, paired | Held-out | Tool selection | Arg exact match | Invalid calls | Steps | Latency / task | Tokens / task (in + out) | BFCL |
  21. |---|---|---|---|---|---|---|---|---|---|---|
  22. | Qwen3-4B-Instruct-2507, prompted (Ollama Q4_K_M) | **87.0** (83.0–90.7) | – | 90.7 | 79.7 | 78.6 | 0.3 | 2.19 | 5.5 s | 5070 + 112 | 86.8* |
  23. | Qwen3-4B-Instruct-2507, prompted (HF 4-bit, same setup as fine-tunes) | not run yet | | | | | | | | | |
  24. | + SFT (Hermes function-calling data) | not run yet | | | | | | | | | |
  25. | + SFT + GRPO | not run yet | | | | | | | | | |
  26. | Qwen3-8B, prompted (Ollama Q4_K_M) | **82.0** (77.7–86.3) | −5.0 (−9.7 to −0.3) vs 4B | 68.0 | 68.3 | 66.2 | 0.0 | 1.95 | 5.4 s | 4442 + 62 | – |
  27. | Qwen3-8B, prompted (HF 4-bit, same setup as fine-tunes) | not run yet | | | | | | | | | |
  28. | API model (optional, any OpenAI-compatible endpoint) | skipped (no key) | | | | | | | | | |
  29. The Δ column is a paired bootstrap over the same tasks. The fine-tuned rows will be compared with the
  30. HF 4-bit base, not the GGUF one, so quantization differences don't masquerade as training gains.
  31. \*BFCL for the local row is a 50-per-category subset (250 questions: simple 96, multiple 90, parallel 70,
  32. parallel-multiple 86, irrelevance 92). The notebook rows run all ~1,240 single-turn questions. The Ollama 8B
  33. row was measured once on the laptop; the 8B comparison now lives in the SFT notebook (HF 4-bit row), because
  34. a 16 GB Mac shouldn't hold an 8B model next to everything else.
  35. **By task type (test / held-out):**
  36. | Model | single | chain | parallel | no tool | refuse | clarify |
  37. |---|---:|---:|---:|---:|---:|---:|
  38. | n (test / held-out) | 103 / 25 | 80 / 72 | 51 / 22 | 16 / 9 | 26 / 12 | 24 / 10 |
  39. | Qwen3-4B | 83.5 / 80.0 | 90.0 / 87.5 | 96.1 / 100 | 87.5 / 100 | 100 / 100 | 58.3 / 100 |
  40. | Qwen3-8B | 76.7 / 84.0 | 67.5 / 47.2 | 100 / 77.3 | 100 / 88.9 | 84.6 / 100 | 100 / 100 |
  41. | + SFT, + SFT + GRPO | not run yet | | | | | |
  42. **By number of gold tool steps (test):** Qwen3-4B 89.1 / 87.3 / 84.4 / 100 for 0 / 1 / 2 / 3+ steps
  43. (n = 46 / 150 / 96 / 8); Qwen3-8B 93.5 / 83.3 / 72.9 / 100.
  44. **Behaviour (test):** correct refusals 100% (4B) vs 84.6% (8B); correct clarifications 58.3% vs 100%; the 4B
  45. answers with the requested `Answer:` line 93% of the time.
  46. **GRPO reward curve:** not run yet. `notebooks/colab_grpo.ipynb` writes `results/grpo/log_history.json`,
  47. and `make report` turns it into `results/figures/reward_curve.png`.
  48. ## What the baselines get wrong
  49. From [evals/error_analysis.md](evals/error_analysis.md), which covers all 155 failures of both baselines:
  50. - **False "I don't have access" refusals are the biggest failure for both models** (31 of the 4B's 53 failures,
  51. 33 of the 8B's 102). They cluster on lookups touching personal data, such as a customer's email, city or
  52. sign-up date. The model never calls `get_customer`, though it calls `get_order` without hesitation. It reads
  53. as over-generalised privacy caution, and it's exactly what the GRPO `behavior` reward penalises.
  54. - **The 8B drops steps in chains.** It refunds the 3.50 EUR return fee instead of "item minus fee", divides
  55. days by 7 in its head and gets it wrong, passes an order id as `customer_id`, invents `customer_id=12345`, or
  56. narrates "let me look up her customer ID" and stops. That accounts for 49 of its failures and is why it
  57. loses to the 4B.
  58. - **The 4B acts when it should ask.** Given "Save a note titled 'Reminder 69'", it saves an empty note.
  59. Asked to refund "Marta Ferreira's order" (she has several), it refunds one of them in full.
  60. ## What surprised me
  61. - **The bigger model lost.** Qwen3-8B (thinking off) asks better clarifying questions but is worse at
  62. following a chain through to the end. It also makes fewer tool calls per task (1.12 vs 1.27), so it isn't
  63. slower despite twice the parameters.
  64. - **My checker had bugs that only reading failures revealed.** "Of products 105, 110 and 140, which has the
  65. largest stock?" -> "Answer: 140" was scored wrong because the gold was the product's *name*. 11 of the 4B's
  66. original 27 held-out failures were this. Clarifications without a question mark ("Please provide the
  67. title and body of the note") and declines phrased as policy ("...is against shop policy") were also missed.
  68. I fixed all three with regression tests and re-scored every run by replaying the recorded outputs (no
  69. regeneration). That moved 33 of 900 episodes from fail to pass and none back. A GRPO run on the buggy checker
  70. would have trained the model to rephrase correct answers. Of everything in this project, that's the reward
  71. hacking I'd worry about most, and it was on my side.
  72. - **The leakage test caught a real leak on its first run.** Train and eval use disjoint customers and products,
  73. but following a train order to its customer leaked eval customers into training tool outputs. Orders now
  74. belong to a pool only if their customer *and* all their products do.
  75. - **Prompt drift hides in small places.** TRL renders environment methods sorted by name; my harness used
  76. definition order. Same tools, different prompt. The tools are sorted now, and a test checks that TRL's
  77. rendering is byte-identical to the harness's. A second case came from Unsloth's model mirror, whose chat
  78. template differs from Qwen's. All 1,060 generation prompts I checked render identically, so the evals were
  79. unaffected. But when rendering a whole conversation it puts an empty `<think></think>` before the final reply,
  80. so the SFT run trained on that. And TRL, which recognises templates by exact match, refused to start GRPO.
  81. Every loader now restores the original template (`train/model.py`), with a test for the GRPO failure.
  82. - **The GGUF 4B garbles identifiers in long multi-call outputs.** In BFCL's parallel category the Q4_K_M
  83. 4B writes `calculate_electric_field_Strength`, `get_protein_ sequence`, `number-2` or broken JSON in the
  84. third or fourth call of a list, after getting the first ones right: 9 of its 15 parallel failures are a
  85. misspelled function name, an unknown argument key or unparseable JSON.
  86. The HF 4-bit rows from the notebook will show whether this is the GGUF quantization or the model.
  87. - **11% of the public SFT data fails strict parsing**, mostly Python-dict quoting inside `<tool_call>`. I drop it
  88. rather than repair it, so SFT doesn't teach a format the parser rejects.
  89. Reward hacking during GRPO itself: not run yet. The reward design anticipates the obvious hacks (shotgun
  90. numbers, repeated `Answer:` lines, naming every option, refuse-everything, ask-everything, claiming a refund
  91. without making one, made-up ids, tool spam), and each has a test in `tests/test_rewards.py`. The GRPO notebook
  92. logs sampled rollouts to `results/grpo/samples.jsonl` so I can check what actually happens.
  93. ## How it works
  94. - `toolenv/`: `ToolEnv`, whose public methods are the tools. That's the shape TRL's `environment_factory`
  95. expects (one instance per rollout), and the eval loop and the MCP server (`mcp_server.py`, MCP SDK 2.3) use the
  96. same methods. Arguments are validated strictly (`"1005"` is not an int) and errors come back as JSON, not
  97. exceptions.
  98. - `tasks/`: 38 training templates and 13 held-out templates. Each one runs its gold plan against the
  99. environment and takes the answer from the tool outputs. Answers are checked by kind (number, text with
  100. closed-domain choices, date, all-of, refuse, clarify), and writes are checked against the environment's final state.
  101. - `agent/`: strict `<tool_call>` parser, a loop that feeds tool errors and parse errors back, and backends.
  102. Ollama runs in raw mode with the model's own Hugging Face chat template, so every backend sees the same prompt.
  103. - `train/`: `rewards.py` (correctness and behaviour are the only positive terms; format, hallucination and
  104. efficiency are penalties only), `sft.py` (Hermes function-calling data, completion-only loss), `grpo.py`
  105. (TRL 1.13 + Unsloth, multi-turn rollouts with real tool execution).
  106. - `evals/`: harness (bootstrap CIs, slices, paired deltas), BFCL v4 subset with a simplified AST checker, error
  107. taxonomy, report/figures.
  108. - `serve/`: FastAPI `/chat` running the agent with tools executed over MCP (stdio), plus a Gradio demo at `/demo`.
  109. ## Running it
  110. Ollama runs natively (Metal). On a 16 GB machine, start it with one model and a 4k context:
  111. `OLLAMA_MAX_LOADED_MODELS=1 OLLAMA_NUM_PARALLEL=1 OLLAMA_CONTEXT_LENGTH=4096 ollama serve` (a 4B Q4 model then
  112. uses ~3.2 GB; the longest prompt in any local run is 3.4k tokens).
  113. ```bash
  114. make setup # uv sync + ollama pull qwen3:4b-instruct-2507-q4_K_M
  115. make test # no model download or GPU needed
  116. make eval NAME=base-4b # test subset + held-out set through Ollama, ~40 min on an M4
  117. make bfcl NAME=base-4b # 50 questions per BFCL category (full set runs in the notebooks)
  118. make report # README tables, figures, error taxonomy
  119. make serve # http://localhost:8000/chat and /demo
  120. ```
  121. `docker build -t tool-calling-grpo . && docker run -p 8000:8000 --add-host=host.docker.internal:host-gateway tool-calling-grpo`
  122. runs the API in a container against Ollama on the host. The API and MCP server use about 0.6 GB together, so a
  123. small Colima VM (`colima start --memory 2`) is enough; don't run a default 8 GB VM next to a local model on a 16 GB Mac. Training runs in `notebooks/colab_sft.ipynb`, then
  124. `notebooks/colab_grpo.ipynb`, on a free Colab or Kaggle T4. Each notebook clones the repo, installs pinned
  125. versions, trains, evaluates and zips `results/` for download.
  126. ## Limitations
  127. - The environment is synthetic and narrow. The test split is in-distribution by construction, so the held-out
  128. set (150 tasks) and BFCL are the only generalisation checks, and both are small.
  129. - Some slices are tiny (clarify: 24 test / 10 held-out), so read per-type numbers with their n.
  130. - Baselines are Q4_K_M GGUF on Ollama, while fine-tunes are evaluated in 4-bit transformers. The notebook
  131. re-evaluates the base model in the fine-tunes' setup for that reason.
  132. - The answer checker is strict and rule-based. I read every baseline failure and fixed what I found, but it will
  133. still under-credit some unusually phrased correct answers. Format compliance is reported separately.
  134. - The BFCL checker is my own simplified reimplementation of its AST matching. Use it to compare rows here, not
  135. against the leaderboard.
  136. - One seed per configuration so far. TODO: repeat the GRPO run with 3 seeds to get variance on the training
  137. effect itself.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.