Skip to content
Back to search

nimble

Local typed decisions, contrastive data curation, and model evaluation.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/bespokelabsai/nimble with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/bespokelabsai/nimble'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/bespokelabsai--nimble/badge.svg)](/listings/bespokelabsai--nimble)

HTML

<a href="/listings/bespokelabsai--nimble"><img src="/api/listings/bespokelabsai--nimble/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Clone the repository and move into its folder. Run all the commands below from this folder.

git clone https://github.com/bespokelabsai/nimble.git nimble
cd nimble

Use Python 3.12. To run the model on a Mac, you need Apple Silicon. Python must also run directly on macOS so that it can use Metal, which is Apple's interface to the GPU. To run the model on Linux, you need an NVIDIA GPU that supports BF16, a 16-bit number format.

Without quantization, the 9B weights alone take about 18 GB. Quantization means storing the weights with fewer bits to save memory. The model needs more memory than this while it runs. The merge step below runs on the CPU. It needs extra RAM, and it needs disk space for both the base weights and the merged weights. A Mac with 64 GB of memory has more free memory for this than a machine with 24 GB.

Download the model

Create a Python environment for preparing the model. On Linux, you can also use this environment to run the model on the GPU. If you do, install a build of PyTorch with CUDA support that works with your GPU driver.

python3.12 -m venv .cache/venvs/nimble
source .cache/venvs/nimble/bin/activate
python -m pip install torch==2.8.0 -r requirements/training.txt

The following accepts either a full checkpoint or a PEFT LoRA adapter. For an adapter, it downloads the pinned base and merges the trained weights once. It records the resolved revision and local model path for both platform examples. No TypeSafe or generation API key is needed for local inference.

python - <<'PYTHON'
import hashlib
import json
from pathlib import Path

from huggingface_hub import snapshot_download

repo = "bespokelabs/Bespoke-Nimble-9B"  # Or "bespokelabs/Bespoke-Nimble-9B-v2"
snapshot = Path(snapshot_download(repo, cache_dir=".cache/huggingface/hub"))
contract_file = snapshot / "schema_config.json"
contract = json.loads(contract_file.read_text()) if contract_file.exists() else {}
if contract:
    from transformers import AutoTokenizer
    from nimble.training.candidate_schema import validate_contract
    validate_contract(contract, AutoTokenizer.from_pretrained(snapshot))

model_path = snapshot
if (snapshot / "adapter_config.json").exists():
    import torch
    from peft import PeftModel
    from transformers import AutoTokenizer, Qwen3_5ForConditionalGeneration

    # The adapter release must include its pinned base and prompt contract.
    base = Qwen3_5ForConditionalGeneration.from_pretrained(
        contract["model"], revision=contract["revision"],
        dtype=torch.bfloat16, device_map="cpu",
    )
    adapter = PeftModel.from_pretrained(base, snapshot)
    merged = adapter.merge_and_unload(safe_merge=True)
    model_path = Path(".cache/models") / ("nimble-9b-" + snapshot.name)
    merged.save_pretrained(model_path)
    (model_path / "schema_config.json").write_text(json.dumps(contract, indent=2))
    AutoTokenizer.from_pretrained(snapshot).save_pretrained(model_path)
    # Preserve adapter identity for automatic temperature selection after merging.
    (model_path / "READY.json").write_text(json.dumps({
        "model": repo, "revision": snapshot.name,
        "adapter_sha256": hashlib.sha256(
            (snapshot / "adapter_model.safetensors").read_bytes()
        ).hexdigest(),
    }, indent=2))

config = {
    "model_path": str(model_path.resolve()),
    "model_id": repo,
    "revision": snapshot.name,
    "max_input_tokens": contract.get("max_length", 2048),
}
Path(".cache/nimble-model.json").write_text(json.dumps(config, indent=2))
print("Ready:", model_path)
PYTHON

Mac with Apple Silicon (MLX)

Use a separate MLX environment after the model preparation step:

deactivate
python3.12 -m venv .venv-mlx
source .venv-mlx/bin/activate
python -m pip install -r requirements/mlx.txt

In Python, pass the saved model settings to `ParallelScorer` so that it loads the prepared 9B weights.

import json
from pathlib import Path

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 15 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "bespokelabsai",
  3. "description": "Agent skills from Bespoke Labs.",
  4. "owner": {
  5. "name": "Bespoke Labs",
  6. "url": "https://bespokelabs.ai"
  7. },
  8. "plugins": [
  9. {
  10. "name": "nimble",
  11. "description": "Build with the hosted Nimble API: typed questions about text or JSON, answered with probabilities.",
  12. "source": "./plugins/nimble"
  13. }
  14. ]
  15. }
README.md · README · 736 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # Bespoke Nimble
  2. **Data, Model, Recipe for an open Jev**
  3. <a href="https://trendshift.io/repositories/247294?utm_source=trendshift-badge&amp;utm_medium=badge&amp;utm_campaign=badge-trendshift-247294" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/trendshift/repositories/247294/daily?language=Python" alt="bespokelabsai%2Fnimble | Trendshift" width="250" height="55"/></a>
  4. #4 repository of the day on Trendshift.
  5. [Model](https://huggingface.co/bespokelabs/Bespoke-Nimble-9B) · [Timeline](#timeline) · [Updates](#updates) · [Capabilities](#capabilities) · [Quickstart](#quickstart) · [Methodology](#methodology) · [Documentation and development](#documentation-and-development) · [Citation](#citation)
  6. ![Introducing Bespoke Nimble. Serving reads the prompt once and then scores one answer token per question. Data curation changes one fact so that the correct answer flips. Training fine-tunes Qwen3.5-9B with LoRA on the answer tokens only. On 324 held-out examples, Bespoke-Nimble-9B matches 90.1% of the reference labels, compared with 66.4% for its base model and 93.2% for Jev 1.13.0.](assets/diagrams/nimble-infographic.svg)
  7. Nimble takes some text and a schema, and makes typed decisions about the text.
  8. The schema is the list of questions to answer. Each question is either a choice
  9. from a list that you give or a true or false question. For each question, Nimble
  10. returns the answer it picked and the probability of each allowed answer.
  11. Nimble makes each decision in one step and does not write out any reasoning
  12. first, so it is fast (blazing fast!). Nimble is
  13. inspired by the System One approach of
  14. [TypeSafe's Jev](https://docs.typesafe.ai/primitives/choice). In this repository,
  15. we share our recipe for training such a model.
  16. Note that we did not distill from Jev. The point of the repository is to show how to curate data, how to train, and to serve such a model, and encourage more research!
  17. You can run [Bespoke-Nimble-9B](https://huggingface.co/bespokelabs/Bespoke-Nimble-9B)
  18. on a Mac with Apple Silicon or on a machine with an NVIDIA GPU.
  19. ## Timeline
  20. These are the main steps from the release of Jev to the launch of Nimble-v4.1.
  21. - On September 14, 2026, TypeSafe released
  22. [Jev](https://docs.typesafe.ai/primitives/choice), a decision model.
  23. - On September 18, 2026, we released Nimble-v1 as
  24. [Bespoke-Nimble-9B](https://huggingface.co/bespokelabs/Bespoke-Nimble-9B),
  25. together with its data and its training recipe.
  26. - On September 29, 2026, [Ollama](https://ollama.com) added Nimble-v2
  27. ([Bespoke-Nimble-9B-v2](https://huggingface.co/bespokelabs/Bespoke-Nimble-9B-v2)),
  28. so you can run it on your own computer with `ollama pull nimble`.
  29. - On October 6, 2026, [Nimble-v3](https://huggingface.co/bespokelabs/Bespoke-Nimble-9B-v3)
  30. was the #1 open model under 25B on the Decision Index. Jev is the only model
  31. with a higher score in this view, and Jev is not open.
  32. <img src="assets/timeline/nimble-v3-decision-index-under-25b.png" alt="The Decision Index chart with the size filter set to models up to 24.4B. Jev, the reference model, scores 60.1. Bespoke Nimble 9B v3 scores 54.7, the highest score of the open models. The next model, Nace.AI Drex v1.5, scores 50.8." width="520">
  33. - On October 13, 2026, we launched Nimble-v4 (9B) and Nimble-v4-max (27B) on
  34. the API at [console.bespokelabs.ai](https://console.bespokelabs.ai). On the
  35. public suite of the Decision Index, Nimble-v4-max is #1 with 68.46, and
  36. Nimble-v4 is #1 of 104 models under 25B with 63.70. Jev 1.13 scores 57.96, and
  37. Perplexity Decider v1.1 scores 62.25. We measured the scores for Nimble-v4 and
  38. Nimble-v4-max in our own run on October 12, 2026. They are not yet on the
  39. Decision Index dashboard.
  40. - On November 5, 2026, we released Nimble-v4.1 (27B). At release, it was tied
  41. for #2 on the Decision Index. It scores 70.0 overall and 72.74 on the public
  42. suite, up from 68.46 on the public suite for Nimble-v4-max. It is available
  43. on the API as `nimble-v4.1-max`, with zero data retention, and
  44. `nimble-latest` now points to it. The weights are on Hugging Face as the
  45. `v4.1` tag of [Nimble-v4-max](https://huggingface.co/bespokelabs/Nimble-v4-max).
  46. <img src="assets/timeline/nimble-v4.1-decision-index.png" alt="The Decision Index chart at the release of Nimble-v4.1. Bespoke Nimble v4.1 Max (27B) scores 70.0 and is in the group tied for #2, with Turion Jev 31B at 70.5 and vLLM-SR d3-ultra at 70.2. The tied #1 group scores 71.1 to 71.9." width="360">
  47. ## Updates
  48. - September 24, 2026: [Bespoke-Nimble-9B](https://huggingface.co/bespokelabs/Bespoke-Nimble-9B) now contains the latest checkpoint, with an 8,192-token context and up to 255 choices per field. Its default is T=1.0; the original release is preserved under the `original-2676` tag, and the separate v2 repository is unchanged. Update this checkout before loading the new release.
  49. - On September 22, 2026, we fitted a temperature for Bespoke-Nimble-9B. With
  50. this temperature, the probabilities better match how often the answers are
  51. right. The model picks the same answers as before. Noul probabilities and
  52. Score values do change, so if you compare them with a threshold, test the
  53. threshold again. See [Probability temperature](#probability-temperature) and
  54. [PR #7](https://github.com/bespokelabsai/nimble/pull/7).
  55. - On September 20, 2026, we published the 2,676 training examples and the 324
  56. held-out examples for Bespoke-Nimble-9B. We had left them out of the first
  57. release by mistake. See the [dataset guide](docs/DATASET.md) and
  58. [PR #5](https://github.com/bespokelabsai/nimble/pull/5).
  59. - On September 19, 2026, we raised the prompt limit of the hosted API to 8,192
  60. tokens for each question. The model was trained on prompts of up to 2,048
  61. tokens, so shorter prompts are better tested. See the
  62. [SGLang deployment guide](docs/MODAL_SERVING.md) and
  63. [PR #4](https://github.com/bespokelabsai/nimble/pull/4).
  64. - On September 18, 2026, [Edgar Dyck](https://github.com/eddited17) added a
  65. public benchmark suite. With it, you can run Bespoke-Nimble-9B and Jev on the
  66. same records from 13 public subsets with human labels. The
  67. [public benchmarks guide](docs/PUBLIC_BENCHMARKS.md) has the steps and the
  68. results. See [PR #2](https://github.com/bespokelabsai/nimble/pull/2).
  69. ## Capabilities
  70. We built Nimble in one day, so expect some rough edges. What Nimble can do comes
  71. from two sources: the first is the base model, Qwen3.5-9B, the second is our
  72. training data, which we curated for a few specific domains.
  73. ### What you can build
  74. | Task | You define | You get back |
  75. | --- | --- | --- |
  76. | Route a request | The destinations and when each one applies | The chosen destination and the probability of each destination |
  77. | Check a condition | A yes or no question and the evidence | True or false, and the probability of each |
  78. | Apply a policy | The rules and the allowed outcomes | A typed decision based on the text you supply |
  79. | Rate an outcome | Ordered levels, each with clear criteria | The chosen level and the probability of each level |
  80. You supply a context, which is the text to judge, and a schema. The schema must
  81. be flat, which means that it has no nested fields. Each field is an enum or a
  82. boolean. An enum field has a fixed list of string choices, and a boolean field
  83. is true or false.
  84. Each allowed answer has a code that is one token long. The scorer reads the
  85. model's logits for these codes. Logits are the raw scores that the model gives
  86. to each token. The scorer turns the logits into probabilities with the softmax
  87. function. Our Python code then builds the output from these probabilities, so
  88. there is no generated JSON to parse. If a field is an ordered rating scale, your
  89. application can use the probabilities to calculate an expected level.
  90. On a Mac, `ParallelScorer` processes the shared context once and then scores all
  91. the fields in parallel. The CUDA scorer scores each field on its own, with the
  92. full prompt each time. Both scorers return the typed output. They also return
  93. the logits and the probabilities of the candidate answers. Each field is scored
  94. separately, so one field cannot see the answer to another field.
  95. ### What you cannot build with the current release
  96. - Nimble accepts only text. You cannot use it to judge other kinds of input,
  97. e.g., images. This is true even though the base model includes a vision part.
  98. - Nimble only picks from the answers you supply. It cannot write text of its
  99. own, e.g., an explanation. It also cannot return nested JSON or a piece of
  100. text taken from the context. An enum field can have 1 to 255 string choices in the latest release,
  101. and a boolean field has two.
  102. - The probabilities are not a guarantee that an answer is correct. Nimble scales
  103. them so that they add up to 1 across the answers you supplied. The latest checkpoint uses T=1.0 and has not had a separate temperature fit.
  104. Earlier releases have their own temperature settings. See [Probability temperature](#probability-temperature). A probability
  105. of 0.9 still does not mean that the answer is right 90% of the time on your
  106. data. If it is possible that none of your answers fit, add an answer that
  107. means "no match". Test any probability threshold on your own data before you
  108. rely on it.
  109. - Each prompt can have at most 8,192 tokens in the latest release. This limit includes the schema and
  110. the part of the prompt that names the field to score. Nimble rejects longer
  111. prompts. Fields cannot depend on each other, so your code must check that the
  112. answers to different fields are consistent.
  113. Nimble’s performance depends on the curated data and domains represented in its training; test it on your own tasks.
  114. But we do see that Nimble is overall better than its base model Qwen3.5-9B in new domains.
  115. ## Quickstart
  116. Clone the repository and move into its folder. Run all the commands below from
  117. this folder.
  118. ```sh
  119. git clone https://github.com/bespokelabsai/nimble.git nimble
  120. cd nimble
  121. ```
  122. Use Python 3.12. To run the model on a Mac, you need Apple Silicon. Python must
  123. also run directly on macOS so that it can use Metal, which is Apple's interface
  124. to the GPU. To run the model on Linux, you need an NVIDIA GPU that supports
  125. BF16, a 16-bit number format.
  126. Without quantization, the 9B weights alone take about 18 GB. Quantization means
  127. storing the weights with fewer bits to save memory. The model needs more memory
  128. than this while it runs. The merge step below runs on the CPU. It needs extra
  129. RAM, and it needs disk space for both the base weights and the merged weights.
  130. A Mac with 64 GB of memory has more free memory for this than a machine with
  131. 24 GB.
  132. ### Download the model
  133. Create a Python environment for preparing the model. On Linux, you can also use
  134. this environment to run the model on the GPU. If you do, install a build of
  135. PyTorch with CUDA support that works with your GPU driver.
  136. ```sh
  137. python3.12 -m venv .cache/venvs/nimble
  138. source .cache/venvs/nimble/bin/activate
  139. python -m pip install torch==2.8.0 -r requirements/training.txt
  140. ```
  141. The following accepts either a full checkpoint or a PEFT LoRA adapter. For an adapter, it downloads the pinned base and merges the trained weights once. It records the resolved revision and local model path for both platform examples. No TypeSafe or generation API key is needed for local inference.
  142. ```sh
  143. python - <<'PYTHON'
  144. import hashlib
  145. import json
  146. from pathlib import Path
  147. from huggingface_hub import snapshot_download
  148. repo = "bespokelabs/Bespoke-Nimble-9B" # Or "bespokelabs/Bespoke-Nimble-9B-v2"
  149. snapshot = Path(snapshot_download(repo, cache_dir=".cache/huggingface/hub"))
  150. contract_file = snapshot / "schema_config.json"
  151. contract = json.loads(contract_file.read_text()) if contract_file.exists() else {}
  152. if contract:
  153. from transformers import AutoTokenizer
  154. from nimble.training.candidate_schema import validate_contract
  155. validate_contract(contract, AutoTokenizer.from_pretrained(snapshot))
  156. model_path = snapshot
  157. if (snapshot / "adapter_config.json").exists():
  158. import torch
  159. from peft import PeftModel
  160. from transformers import AutoTokenizer, Qwen3_5ForConditionalGeneration
  161. # The adapter release must include its pinned base and prompt contract.
  162. base = Qwen3_5ForConditionalGeneration.from_pretrained(
  163. contract["model"], revision=contract["revision"],
  164. dtype=torch.bfloat16, device_map="cpu",
  165. )
  166. adapter = PeftModel.from_pretrained(base, snapshot)
  167. merged = adapter.merge_and_unload(safe_merge=True)
  168. model_path = Path(".cache/models") / ("nimble-9b-" + snapshot.name)
  169. merged.save_pretrained(model_path)
  170. (model_path / "schema_config.json").write_text(json.dumps(contract, indent=2))
  171. AutoTokenizer.from_pretrained(snapshot).save_pretrained(model_path)
  172. # Preserve adapter identity for automatic temperature selection after merging.
  173. (model_path / "READY.json").write_text(json.dumps({
  174. "model": repo, "revision": snapshot.name,
  175. "adapter_sha256": hashlib.sha256(
  176. (snapshot / "adapter_model.safetensors").read_bytes()
  177. ).hexdigest(),
  178. }, indent=2))
  179. config = {
  180. "model_path": str(model_path.resolve()),
  181. "model_id": repo,
  182. "revision": snapshot.name,
  183. "max_input_tokens": contract.get("max_length", 2048),
  184. }
  185. Path(".cache/nimble-model.json").write_text(json.dumps(config, indent=2))
  186. print("Ready:", model_path)
  187. PYTHON
  188. ```
  189. ### Mac with Apple Silicon (MLX)
  190. Use a separate MLX environment after the model preparation step:
  191. ```sh
  192. deactivate
  193. python3.12 -m venv .venv-mlx
  194. source .venv-mlx/bin/activate
  195. python -m pip install -r requirements/mlx.txt
  196. ```
  197. In Python, pass the saved model settings to `ParallelScorer` so that it loads
  198. the prepared 9B weights.
  199. ```python
  200. import json
  201. from pathlib import Path
  202. from nimble.scoring.parallel_scorer import ParallelScorer
  203. config = json.loads(Path(".cache/nimble-model.json").read_text())
  204. scorer = ParallelScorer(**config)
  205. ```
  206. > [!NOTE]
  207. > If you call `ParallelScorer()` with no arguments, it loads the Qwen3.5-4B
  208. > model that we used as a baseline, not Nimble. The MLX runner cannot load a
  209. > LoRA adapter folder directly, so use the merged folder that you prepared
  210. > above. The MLX runner also does not support quantized weights.
  211. ### Linux with an NVIDIA GPU (CUDA)
  212. Activate the environment that you used to prepare the model. Then check that
  213. PyTorch can use the GPU and that the GPU supports BF16.
  214. ```sh
  215. source .cache/venvs/nimble/bin/activate
  216. python -c 'import torch; assert torch.cuda.is_available() and torch.cuda.is_bf16_supported()'
  217. ```
  218. In Python, load the same prepared weights.
  219. ```python
  220. import json
  221. from pathlib import Path
  222. from nimble.scoring.cuda_scorer import CudaCandidateScorer
  223. config = json.loads(Path(".cache/nimble-model.json").read_text())
  224. scorer = CudaCandidateScorer(**config)
  225. ```
  226. You can also run the adapter without merging it, in the same way as our
  227. training evaluation. See the [training guide](docs/NIMBLE_TRAINING.md) for how
  228. to do this. The CUDA runner scores each field with the full prompt. So it
  229. processes the shared context again for each field, while the MLX runner
  230. processes it only once.
  231. ### Make a typed decision
  232. After you load either scorer, continue in the same Python session.
  233. ```python
  234. schema = {
  235. "priority": {
  236. "type": "enum",
  237. "choices": ["HIGH", "LOW"],
  238. "description": "Urgency based on current business impact.",
  239. "choice_descriptions": {
  240. "HIGH": "A critical business operation is currently blocked.",
  241. "LOW": "An optional enhancement with no current business impact.",
  242. },
  243. },
  244. "requires_review": {
  245. "type": "boolean",
  246. "description": "Whether customers are unable to complete a purchase.",
  247. },
  248. }
  249. result = scorer.score("The payment service is down for all customers.", schema)
  250. print(result["output"])
  251. print(result["fields"]["priority"]["scores"]) # Candidate probabilities.
  252. print(result["fields"]["priority"]["logits"])
  253. ```
  254. The output has this shape. The values depend on the model:
  255. ```json
  256. {"priority": "HIGH", "requires_review": true}
  257. ```
  258. Load the scorer once and reuse it for each new context. The latest checkpoint
  259. uses T=1.0; the following fitted temperature applies only to the original release. For revision
  260. `93ec5d6ff1a9cd31d6cc0e0c58d312465d36de7c` of Bespoke-Nimble-9B, both scorers
  261. use a temperature of 2.179 by default. The scorers need this full revision hash,
  262. and a short hash such as `93ec5d6` does not match. The download step above saves
  263. the full hash in `.cache/nimble-model.json`. We fitted this temperature so that
  264. the probabilities better match how often the answers are right. See
  265. [Probability temperature](#probability-temperature). For other models, the
  266. default temperature is `1.0`, except for
  267. [`Bespoke-Nimble-9B-v2`](https://huggingface.co/bespokelabs/Bespoke-Nimble-9B-v2):
  268. v2 automatically uses **2.179078721266035** in both scorers, including local merges
  269. whose `READY.json` identifies its adapter. This is a transferred release default,
  270. not a temperature fitted independently for v2. Documentation-only Hub revisions
  271. retain this default. Deliberately using raw v2 probabilities requires both
  272. `temperature=1.0` and `allow_uncalibrated=True`; passing only `temperature=1.0`
  273. raises an error. Other positive temperatures remain explicit overrides. See the
  274. [scoring guide](docs/PARALLEL_SCORING.md) for the schema rules and the ways you
  275. can run the scorer.
  276. ## Methodology
  277. This section documents the original release’s data and recipe. The latest checkpoint is identified in [Updates](#updates).
  278. The serving methodology and training data curation are heavily inspired by [Bespoke-MiniCheck](https://huggingface.co/bespokelabs/Bespoke-MiniCheck-7B).
  279. ### Serving
  280. We follow the approach laid out by [Niels Rogge](https://x.com/NielsRogge): see this [post on how Jev does decoding](https://x.com/NielsRogge/status/2100239244501430438).
  281. In a nutshell:
  282. * Process the context and schema once (prefill the KV-cache)
  283. * We obtain the scores for the tokens we care about.
  284. We used this approach (we didn't have to do kv-cache prefill) in Bespoke-MiniCheck, since it always returned a single json tuple: `{"is_claim_supported_by_context": p}`.
  285. ### Contrastive data curation
  286. The challenge here is that we don't have access to probabilities from a teacher model (or humans). But as you saw above, we just need to somehow get the logits and ensure the logits are as calibrated to estimate the probabilities as possible.
  287. We push the model to be calibrated to be a better decision maker by creating negative examples, which forces the model to become a better discriminator.
  288. So we made the training data with a new method that we call contrastive data
  289. curation. In this method, we write two examples that are almost the same. They
  290. differ in one relevant fact, and this difference changes the correct answer.
  291. Everything else stays the same, including the question and the policy. From
  292. these pairs, the model learns which evidence should change its decision.
  293. [![Watch the contrastive data curation video: changing the signer from Mira to Noah flips the answer from true to false.](assets/video/contrastive-curation/poster.jpg)](assets/video/contrastive-curation/contrastive-curation.mp4)
  294. [Watch the 34-second explanation](assets/video/contrastive-curation/contrastive-curation.mp4) (silent).
  295. In this example, the rule is that a refund is authorized only when its sole
  296. authorization was signed by someone who can approve refunds for that account.
  297. The example has two evidence sentences, and you need both of them to answer.
  298. | Evidence | First example | Changed example |
  299. | --- | --- | --- |
  300. | Who can approve refunds | Only Mira may authorize refunds for account 42. | Unchanged |
  301. | Authorization record | The sole authorization for this refund on account 42 was signed by **Mira**. | The sole authorization for this refund on account 42 was signed by **Noah**. |
  302. | Is the refund authorized? | `true` | `false` |
  303. Our curation pipeline uses this idea for Choice, Noul and Score tasks. It has
  304. four steps:
  305. 1. Check the decision rules. We start from the schema and the policy of a
  306. training source. We write down the basic facts that the rules depend on.
  307. Then we check that the rules can lead to different answers.
  308. 2. Build the pair. We write two evidence sentences, and you need both of them
  309. to find the answer. Then we change at most eight words in one of the
  310. sentences. We make this edit to change one fact, which we call the focus
  311. fact, and so the label changes too. The rest of the context and the other
  312. facts stay the same.
  313. 3. Check both examples. We use separate model calls to check the facts in each
  314. example. We also use these calls to check that the example is consistent
  315. with the policy and that the text does not hint at the answer. Then we
  316. remove each evidence sentence in turn. With either sentence removed, the
  317. focus fact must become unknown, even with all of the other text present.
  318. This way, we know that no other text gives away the answer.
  319. 4. Make the labels. We use code to apply the checked rules to the checked
  320. facts. We keep a pair only when every required check passes and the two
  321. labels differ. We save all model requests and responses, along with the
  322. check results. You can use them to replay the whole process offline.
  323. The examples with an evidence sentence removed are only checks. We do not add
  324. them to the training data with labels, because missing evidence does not mean
  325. that the answer is false or that the score is low. A source family is the group
  326. of examples that we built from one training source. We keep both examples of a
  327. pair, and all examples from one source family, in the same split. The split is
  328. either training or evaluation.
  329. #### Current dataset
  330. There is one training set and one held-out set.
  331. | File | Examples | Use |
  332. | --- | ---: | --- |
  333. | [data/train.jsonl](data/train.jsonl) | 2,676 | Original release training set |
  334. | [data/eval.jsonl](data/eval.jsonl) | 324 | Frozen final evaluation only |
  335. The published model's training data covers **10 subject categories**. The tables
  336. below count only its 2,676 training examples and the 324-example holdout.
  337. Category counts come from each record's `domain` field; the holdout covers six
  338. of these categories.
  339. | Category | Training examples | Held-out examples |
  340. | --- | ---: | ---: |
  341. | Commerce | 242 | 58 |
  342. | Education | 230 | 70 |
  343. | Home | 300 | 0 |
  344. | Media | 256 | 44 |
  345. | Public services | 194 | 106 |
  346. | Science | 300 | 0 |
  347. | Software | 300 | 0 |
  348. | Supply chain | 270 | 30 |
  349. | Travel | 284 | 16 |
  350. | Workplace | 300 | 0 |
  351. | **Total** | **2,676** | **324** |
  352. Across these subjects, each example asks one of three
  353. [typed questions](https://docs.typesafe.ai/primitives):
  354. | Task type | Judgment | Training examples | Held-out examples |
  355. | --- | --- | ---: | ---: |
  356. | Choice | Select one candidate | 856 | 146 |
  357. | Noul (Boolean) | Decide whether a condition holds | 888 | 114 |
  358. | Score | Judge an ordered rubric level | 932 | 64 |
  359. | **Total** | | **2,676** | **324** |
  360. All of the labels are synthetic: a model checked them, and no person has reviewed them. Separate calls to the same model can make the same mistake, so the checks can miss some errors. See the
  361. [generation and replay guide](docs/TRAINING_EVAL_CURATION.md) for the full
  362. method.
  363. The command below checks the data files. It needs no GPU and makes no API
  364. calls. It confirms that the files match their saved checksums and have the
  365. expected number of examples. It also confirms that no example or source family
  366. appears in both the training set and the evaluation set.
  367. ```sh
  368. .venv-curator/bin/python -m nimble.training.verify_dataset
  369. ```
  370. See the [dataset details](docs/DATASET.md) and the
  371. [training guide](docs/NIMBLE_TRAINING.md). The curation guides also describe
  372. older ways that we built data. We no longer keep the source folders from those
  373. older methods.
  374. ### Finetuning
  375. The trainer applies LoRA to Qwen3.5-9B, optimizing cross-entropy over the allowed candidate logits. It learns the typed decision directly. Saved Jev probabilities are available for future soft-target distillation if you need it, but the current training objective uses hard reference labels that are not derived from Jev.
  376. The final recipe uses these settings:
  377. - LoRA rank 16
  378. - Learning rate 5e-5
  379. - Effective batch size 8
  380. - Random seed 17
  381. - One epoch of training
  382. It preserves the three-epoch linear learning rate schedule used during selection, stopping after the selected epoch. Training uses BF16 and a 2,048-token prompt limit. Tuning ran on L40S; the final fit and evaluation ran on H100.
  383. ### Evaluation on 324 held-out examples
  384. ![Reference-label agreement on the same 324 examples: Jev 93.21%, Bespoke-Nimble-9B 90.12%, Qwen3.8-27B 84.88%, Qwen3.5-9B 66.36%, Qwen3.5-4B 61.42%, Qwen3.5-0.8B 45.37%, and Gemma 3 270M IT 28.70%.](assets/evidence-324-comparison.svg)
  385. | Model | Reference matches | Agreement |
  386. | --- | ---: | ---: |
  387. | Gemma 3 270M IT | 93/324 | 28.70% |
  388. | Qwen3.5-0.8B | 147/324 | 45.37% |
  389. | Qwen3.5-4B | 199/324 | 61.42% |
  390. | Qwen3.5-9B | 215/324 | 66.36% |
  391. | Qwen3.8-27B | 275/324 | 84.88% |
  392. | **Bespoke-Nimble-9B** | **292/324** | **90.12%** |
  393. | Jev 1.13.0 | 302/324 | 93.21% |
  394. On these 324 examples, Bespoke-Nimble-9B matched 17 more reference labels than
  395. the untuned 27B model, which is 5.25 percentage points more. Jev matched 10 more
  396. reference labels than Bespoke-Nimble-9B, which is 3.09 points more. The untuned
  397. models are models that we did not fine-tune for this task. We tested all seven
  398. models on the same examples with the same reference labels. We ran Gemma and
  399. the untuned Qwen models on an H100 GPU. For Bespoke-Nimble-9B, we reused
  400. checked results from an earlier H100 run. For Jev, we reused results from an
  401. earlier run through its API.
  402. The reference labels are synthetic. The 324 examples form 162 pairs of closely
  403. related examples. All of them come from only six source families, so this is a narrow
  404. test.
  405. For the untuned models, we computed the scores of the candidate answers in FP32,
  406. a 32-bit number format. For Bespoke-Nimble-9B, we kept the setup that we had
  407. already checked, which computes the output layer in BF16. In a new run of the
  408. untuned 9B model, one answer that had been a 50/50 tie was no longer a tie. This
  409. changed the model's count from 214 to 215. For rating tasks, we count a match
  410. when the most probable level equals the reference level. The
  411. [machine-readable comparison](assets/evidence-324-results.json) contains
  412. the reference-match counts shown above.
  413. We checked the saved adapter in two ways. After we reloaded it, it gave exactly
  414. the same logits as before. When we turned the adapter off, the model gave the
  415. same results as the base model. We ran these checks only on the CUDA path that
  416. loads the adapter without merging it. We did not run a separate quality test on
  417. the merged model that the Mac and Linux quickstarts use. See the
  418. [training guide](docs/NIMBLE_TRAINING.md) for the commands and for the contract
  419. file saved with each model. In the contract file, we record the base model and
  420. the prompt format that the model expects.
  421. For external, human-labeled tests on tasks outside these training categories, see
  422. the [public benchmarks guide](docs/PUBLIC_BENCHMARKS.md), which runs Bespoke-Nimble-9B
  423. and Jev on the same records from thirteen public subsets, starting with VitaminC.
  424. ### Probability temperature
  425. The scorers turn the logits into probabilities with the softmax function. Before
  426. the softmax, they divide the logits by a number that is called the temperature.
  427. With a temperature above 1, the probabilities are less extreme. The order of the
  428. answers by probability is the same at any temperature. So the model picks the
  429. same answer at any temperature. The probabilities themselves do change, and so
  430. does an expected level that you calculate from them. If your code compares a
  431. probability or an expected level with a threshold, test the threshold again.
  432. The latest checkpoint uses T=1.0 without a separate fit. The following results
  433. describe the original release. We fitted one temperature for revision `93ec5d6` of Bespoke-Nimble-9B. We used
  434. two sets of 300 examples, and the two sets come from different sources. None of
  435. these examples are in the training data or in the 324 held-out examples above.
  436. On the first set, we picked the temperature with the lowest log loss. The log
  437. loss is the average negative log of the probability that the model gave to the
  438. correct answer. Then we checked the temperature on the second set. Before the
  439. check, we decided to keep the temperature only if the log loss on the second set
  440. went down and the Brier score did not go up. The fitted temperature is 2.179.
  441. The hosted API and both local scorers use it for this revision.
  442. At a temperature of 1, the probabilities were too high. On the second set, the
  443. average probability of the picked answer was 0.89, but only 73% of the picked
  444. answers were right. The table shows the results at both temperatures. Lower is
  445. better for each measure.
  446. | Examples | Measure | Temperature 1 | Temperature 2.179 |
  447. | --- | --- | ---: | ---: |
  448. | Second set (300) | Expected calibration error | 0.128 | 0.066 |
  449. | Second set (300) | Log loss | 0.692 | 0.555 |
  450. | Second set (300) | Brier score | 0.348 | 0.295 |
  451. | Held-out set (324) | Expected calibration error | 0.052 | 0.054 |
  452. | Held-out set (324) | Log loss | 0.318 | 0.259 |
  453. | Held-out set (324) | Brier score | 0.154 | 0.144 |
  454. For the expected calibration error, we sort the examples into ten groups by the
  455. probability of the picked answer. In each group, we compare the average
  456. probability with how often the picked answer is right. The expected calibration
  457. error is the average of these gaps, weighted by the size of each group. For the
  458. Brier score, the correct answer counts as 1 and each other answer counts as 0.
  459. The Brier score is the sum of the squared gaps between these numbers and the
  460. probabilities, averaged over the examples.
  461. In 24 of the 300 examples in the second set, the correct answer is a set of
  462. probabilities instead of one answer. For these examples, the log loss and the
  463. Brier score compare the model's probabilities with those reference
  464. probabilities. The expected calibration error leaves out these 24 examples. At
  465. both temperatures, the model picked the right answer on 220 of the 300 examples
  466. in the second set and on 292 of the 324 held-out examples.
  467. The results are not better for every kind of question. On the 64 rating
  468. questions in the 324 held-out examples, the probabilities of the picked answers
  469. became too low for how often those answers were right. The expected calibration
  470. error for these questions rose from 0.105 to 0.177.
  471. We fitted the temperature on the CUDA path that loads the adapter without
  472. merging it. The hosted API and the Mac and Linux quickstarts use merged weights.
  473. The logits from merged weights can be slightly different. We did not check the
  474. temperature again on the merged weights.
  475. ### Observed inference latency
  476. In the table below, each time is in milliseconds per example. We calculated
  477. these times from the timing that we saved for each request. Each example has
  478. one question. The local models and Jev return a typed answer and the
  479. probability of each candidate answer. The OpenRouter models generate text in
  480. the usual way, with the reasoning setting at medium. For local scoring, we run
  481. the model once per example. We do not sample more than once, and the model does
  482. not write an explanation.
  483. | Model | Examples | Median (ms) | Mean (ms) | p95 (ms) | Run / dataset |
  484. | --- | ---: | ---: | ---: | ---: | --- |
  485. | Gemma 3 270M IT | 324 | 21.8 | 23.6 | 29.2 | H100 · contrastive holdout |
  486. | Qwen3.5-0.8B | 324 | 48.6 | 49.4 | 60.2 | H100 · contrastive holdout |
  487. | Qwen3.5-4B | 324 | 58.0 | 58.6 | 70.3 | H100 · contrastive holdout |
  488. | Qwen3.5-9B | 324 | 58.1 | 59.8 | 75.8 | H100 · contrastive holdout |
  489. | Qwen3.8-27B | 324 | 145.3 | 145.2 | 185.5 | H100 · contrastive holdout |
  490. | Bespoke-Nimble-9B | 120 | 106.0 | 110.1 | 119.8 | H100 · contrastive holdout|
  491. | Bespoke-Nimble-9B | 324 | 444.0 | 546.0 | 981.0 | M5 Pro 64GB · contrastive holdout|
  492. | Jev 1.13.0 | 324 | 246.7 | 267.0 | 347.4 | TypeSafe API · contrastive holdout |
  493. | DeepSeek-V4.1-Flash | 100 | 2896.4 | 5238.0 | 15065.4 | OpenRouter / Fireworks · general eval |
  494. | Qwen3.8 2.4T A95B | 100 | 2792.3 | 3108.0 | 5110.5 | OpenRouter / Modal · general eval |
  495. ## Documentation and development
  496. | I want to… | Start here |
  497. | --- | --- |
  498. | Define fields or understand parallel scoring | [Scoring guide](docs/PARALLEL_SCORING.md) |
  499. | Compare Nimble and Jev interactively | [Comparison app](docs/COMPARISON_APP.md) |
  500. | Host the published checkpoint on Modal | [SGLang deployment](docs/MODAL_SERVING.md) · [Try the public API](docs/TRY_NIMBLE.md) |
  501. | Understand the retained training and evaluation data | [Dataset guide](docs/DATASET.md) |
  502. | Evaluate on public, human-labeled benchmarks | [Public benchmarks](docs/PUBLIC_BENCHMARKS.md) |
  503. | Create or replay contrastive training data | [Curation guide](docs/TRAINING_EVAL_CURATION.md) |
  504. | Train or use a schema adapter | [Training guide](docs/NIMBLE_TRAINING.md) |
  505. | Build on the hosted Nimble API with a coding agent | [Agent skill](#agent-skill) |
  506. Compare two models only when both were tested on the same examples in the same
  507. way. The guides describe how to generate data and evaluation outputs locally;
  508. these generated files are not committed to Git.
  509. ### Agent skill
  510. The Nimble agent skill helps coding agents build on the hosted Nimble API at
  511. `api.bespokelabs.ai`. It covers writing questions, choosing a model, handling
  512. errors, and acting on the probabilities. It does not cover running the model
  513. from this repository locally. You can [read the skill](plugins/nimble/skills/nimble/SKILL.md)
  514. directly.
  515. To install it in Claude Code:
  516. ```sh
  517. claude plugin marketplace add bespokelabsai/nimble
  518. claude plugin install nimble@bespokelabsai
  519. ```
  520. On Claude Code 2.1.275 or later, `/plugin install nimble --marketplace bespokelabsai/nimble`
  521. does both steps at once. In Claude Code, you can call the skill directly with
  522. `/nimble:nimble`.
  523. To install it in another agent, use [skills.sh](https://skills.sh) and select
  524. your agent when asked:
  525. ```sh
  526. npx skills add bespokelabsai/nimble --skill nimble
  527. ```
  528. To have an agent install the skill for you, give it this prompt:
  529. ```text
  530. Install the Nimble skill. If you're in Claude Code, run `claude plugin marketplace add bespokelabsai/nimble`, then `claude plugin install nimble@bespokelabsai`. If you're in another agent, run `npx skills add bespokelabsai/nimble --skill nimble` and select your agent. Use one installation method. You can read the skill directly at https://github.com/bespokelabsai/nimble/blob/main/plugins/nimble/skills/nimble/SKILL.md (raw: https://raw.githubusercontent.com/bespokelabsai/nimble/main/plugins/nimble/skills/nimble/SKILL.md). Then use the Nimble skill when working on this project.
  531. ```
  532. ### Development
  533. Use a separate Python environment for each of these, because they need
  534. different package versions:
  535. - MLX
  536. - PyTorch
  537. - Curator
  538. The dependency lists are in [requirements/](requirements/). From the project
  539. root, run the offline tests that apply to your change.
  540. ```sh
  541. .venv-mlx/bin/python -m unittest tests.test_parallel_scorer tests.test_evaluate_pilot tests.test_model_evaluation
  542. .venv-curator/bin/python -m unittest tests.test_dataset_io tests.test_diverse_dataset
  543. ```
  544. To measure how much time MLX saves by processing the shared context once for
  545. several fields, run the benchmark for schemas with several fields.
  546. ```sh
  547. .venv-mlx/bin/python -m nimble.evaluation.benchmark_parallel --repeats 3
  548. ```
  549. Each record in the saved dataset evaluations has only one field. So you cannot
  550. use those evaluations to measure the time saved by scoring several fields in
  551. parallel. Their recorded times are also not from a controlled serving test.
  552. ### Folder layout
  553. ```text
  554. nimble/
  555. scoring/ # Local MLX and PyTorch inference
  556. datasets/ # Generation, curation, and TypeSafe labeling
  557. evaluation/ # Quality metrics and inference benchmarks
  558. training/ # Schema-aware CUDA LoRA training
  559. examples/ # Schemas and saved example outputs
  560. data/ # train.jsonl (2,676), eval.jsonl (324), and verification metadata
  561. evaluations/ # Generated locally; not included in Git
  562. assets/ # Comparison graphics, source results, and videos
  563. docs/ # Current usage, training, curation, and deployment guides
  564. ignore/ # Local archive of historical docs and assets; not included in Git
  565. requirements/ # Backend-specific dependencies
  566. tests/ # Offline and small-model checks
  567. plugins/nimble/ # Agent skill for the hosted Nimble API (see Agent skill)
  568. .claude-plugin/ # Marketplace file that installs the agent skill
  569. ```
  570. ## Authors
  571. * Bespoke Labs, Shreyas Pimpalgaonkar, Anirudha Ramesh, Pratik Dalal, Maheswaran Sathiamoorthy
  572. ## Citation
  573. If you use this repository, please cite it. Also give the model revision and
  574. the dataset release that you used.
  575. ```bibtex
  576. @misc{nimble2026,
  577. author = {{Bespoke Labs} and Pimpalgaonkar, Shreyas and Ramesh, Anirudha and Dalal, Pratik and Sathiamoorthy, Maheswaran},
  578. title = {Nimble},
  579. year = {2026},
  580. howpublished = {\url{https://github.com/bespokelabsai/nimble}},
  581. note = {Model: https://huggingface.co/bespokelabs/Bespoke-Nimble-9B}
  582. }
  583. ```
  584. ## Acknowledgments
  585. 1. [TypeSafe](https://typesafe.ai) for making Jev.
  586. 2. [Niels Rogge](https://x.com/NielsRogge) for a [post on how Jev does decoding](https://x.com/NielsRogge/status/2100239244501430438).
  587. 3. [Harsha Gundala](https://x.com/harshagundal) for [inspiring us to work on this](https://x.com/harshagundal/status/2100044305536889015).
  588. 4. [Greg Durett](https://gregdurrett.github.io/) and [Liyan Tang](https://www.tangliyan.com/)'s work on MiniCheck (and check out Bespoke-MiniCheck which we did with them), which was two years early and laid the foundations.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.