Skip to content
Back to search

RSIBench-Data

Synthetic data generation, post-training, and E2B benchmark evaluation infrastructure.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/evolvent-ai--rsibench-data/badge.svg)](/listings/evolvent-ai--rsibench-data)

HTML

<a href="/listings/evolvent-ai--rsibench-data"><img src="/api/listings/evolvent-ai--rsibench-data/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

These steps run a script from the network that findsafeskills does not scan.

1. Install Runtime Prerequisites

- Python 3.12 - A Tinker API key with access to `Qwen/Qwen3.5-35B-A3B-Base` - An E2B API key; the Pro plan is recommended for evaluation workloads - Claude Code CLI as `claude` when `DATA_AGENT_HARNESS=claude-code` - Codex CLI as `codex` when `DATA_AGENT_HARNESS=codex` - Kimi Code CLI as `kimi` when `DATA_AGENT_HARNESS=kimi-code`

Install Claude Code with Anthropic's native installer on macOS, Linux, or WSL:

curl -fsSL https://claude.ai/install.sh | bash
claude --version

Alternative official package-manager installs:

# macOS
brew install --cask claude-code

# Windows
winget install Anthropic.ClaudeCode

Install Codex in a Node/npm environment:

npm install -g @openai/codex
codex --version

Install Kimi Code with Moonshot's installer, or with npm on Node.js 22.19.0 or newer:

curl -fsSL https://code.kimi.com/kimi-code/install.sh | bash
kimi --version

For interactive local Claude Code use, run `claude` once and complete its login flow. Automated RSIBench-Data sessions can instead use repository-local `.env` API configuration, described below.

2. Create the Python Environment

bash tools/setup_env.sh
source .venv/bin/activate

3. Initialize Seed Factories

Seed factories are read-only references used by the data-generation agent. The selected benchmark profile exposes only the factories in its allowlist; a custom profile can opt into all factories with `"seed_factories": ["*"]`.

git submodule update --init --recursive seed_factories

The agent must inspect the exposed references when designing training data.

4. Configure Secrets and Models

cp .env.example .env

Required fields:

TINKER_API_KEY=...
E2B_API_KEY=...

The model names and API-version strings below are **example values**, not a compatibility guarantee. Providers and account entitlements change over time: before starting a paid run, replace them with values supported by your current Tinker, Anthropic, and Codex-compatible endpoint accounts. The repository defaults `TARGET_MODEL` to `Qwen/Qwen3.5-35B-A3B-Base`; the documented session and base-model examples use that same model for consistency. Other Tinker-supported target models (for example a Qwen 3.6 variant, when available to your account) may be supplied explicitly as `TARGET_MODEL`.

Common optional fields:

# Claude Code harness only
ANTHROPIC_BASE_URL=...
ANTHROPIC_API_KEY=...
ANTHROPIC_AUTH_TOKEN=
ANTHROPIC_MODEL=claude-opus-4-8
ANTHROPIC_DEFAULT_OPUS_MODEL=
ANTHROPIC_DEFAULT_SONNET_MODEL=
ANTHROPIC_DEFAULT_HAIKU_MODEL=
ANTHROPIC_DEFAULT_FABLE_MODEL=
CLAUDE_CODE_SUBAGENT_MODEL=
ENABLE_TOOL_SEARCH=
CLAUDE_CODE_AUTO_COMPACT_WINDOW=
CLAUDE_CODE_EFFORT_LEVEL=
CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=

# Codex harness only
CODEX_BASE_URL=...
CODEX_API_KEY=...
CODEX_MODEL=gpt-5.5
CODEX_RESPONSES_API_VERSION=2025-04-01-preview

# Generated rollout/API calls made by data-generation scripts
DATA_AGENT_ROLL_BASE_URL=...
DATA_AGENT_ROLL_API_KEY=...
DATA_AGENT_ROLL_MODEL=claude-opus-4-8

The Claude Code harness uses `ANTHROPIC_BASE_URL`, `ANTHROPIC_MODEL`, and either `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN`. `ANTHROPIC_MODEL` is also the default `DATA_AGENT_MODEL` for the Claude Code orchestrator. Additional Claude Code environment variables can be set in `.env` and are passed through to the Claude subprocess.

Generated data scripts that perform rollout/API calls use `DATA_AGENT_ROLL_BASE_URL`, `DATA_AGENT_ROLL_API_KEY`, and `DATA_AGENT_ROLL_MODEL`.

When `DATA_AGENT_HARNESS=claude-code`, the runner passes the configured Claude Code environment to the Claude subprocess. `ANTHROPIC_API_KEY` mode uses Claude Code's bare/project setting path; `ANTHROPIC_AUTH_TOKEN` mode is used as-is. Both credential fields may be left unset to use the caller's existing Claude CLI configuration.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 731 lines · 1 finding marked

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <h1 align="center">RSIBench-Data</h1>
  2. <p align="center">
  3. <strong>Budget-aware synthetic data generation and post-training for long-horizon agent benchmarks.</strong>
  4. </p>
  5. <div align="center">
  6. [![Website][website-image]][website-url]
  7. [![Evolvent AI][evolvent-image]][evolvent-url]
  8. [![X][x-image]][x-url]
  9. [![LinkedIn][linkedin-image]][linkedin-url]
  10. ![Python 3.12](https://img.shields.io/badge/Python-3.12-3776AB?logo=python&logoColor=white)
  11. ![Training: Tinker](https://img.shields.io/badge/Training-Tinker-111827)
  12. ![Evaluation: Harbor](https://img.shields.io/badge/Evaluation-Harbor-0F766E)
  13. ![Runtime: E2B](https://img.shields.io/badge/Runtime-E2B-F97316)
  14. [![License][license-image]][license-url]
  15. </div>
  16. RSIBench-Data evaluates whether a data-generation agent can improve a target
  17. model by synthesizing benchmark-specific post-training data under a fixed,
  18. real training and evaluation chain. The agent is given a wall-clock budget and
  19. a Tinker cost budget, then iteratively creates training data, launches LoRA SFT,
  20. evaluates the resulting checkpoint, and selects a final submission.
  21. ## System Overview
  22. <p align="center">
  23. <img src="assets/system_overview.png" alt="Tinker training and evaluation pipeline" width="100%">
  24. </p>
  25. Training and sampling run on Tinker. Evaluation runs entirely in E2B cloud
  26. sandboxes through the proxy, so no local Docker runtime is required. The E2B
  27. implementation lives in [`backend/e2b_eval/`](backend/e2b_eval/).
  28. ### Core Properties
  29. - **Closed-loop data optimization**: generate data, train, evaluate, inspect,
  30. and iterate within explicit time and cost budgets.
  31. - **Real post-training**: every attempt uses Tinker LoRA SFT and produces a
  32. sampler checkpoint used by the evaluation proxy.
  33. - **Isolated cloud evaluation**: Harbor agents and task environments run in
  34. per-attempt E2B sandboxes.
  35. - **Multiple orchestrator harnesses**: the same workspace contract supports
  36. Claude Code, Codex, and Kimi Code.
  37. - **Benchmark-specific profiles**: each profile fixes the prompt, dataset,
  38. Harbor agent, evaluation shape, and allowed seed factories.
  39. - **Auditable artifacts**: training inputs, estimates, model usage, Harbor
  40. trajectories, verifier outputs, and final-selection records are retained.
  41. ## Benchmark Profiles
  42. `TARGET_BENCHMARK` is resolved through `benchmarks/*/spec.json`. Each profile
  43. defines its own data-agent prompt, Harbor configuration, official evaluation
  44. shape, and seed-factory allowlist.
  45. | `TARGET_BENCHMARK` | Official dataset | Harbor agent | Tasks | Attempts | Seed factories |
  46. |---|---|---:|---:|---:|---|
  47. | `swe-bench-verified` | Local seed-23 random SWE-bench Verified subset | `mini-swe-agent` | 100 | 1 | SWE-smith, SWE-Gym, swe-factory |
  48. | `swe-bench-multilingual` | Local seed-23 random SWE-bench Multilingual subset | `mini-swe-agent` | 100 | 1 | SWE-smith, SWE-Gym, swe-factory |
  49. | `swe-bench-pro` | Local seed-23 random SWE-bench Pro subset | `mini-swe-agent` | 100 | 1 | SWE-smith, SWE-Gym, swe-factory |
  50. | `terminal-bench-2` | `terminal-bench/terminal-bench-2` | `terminus-2` | 89 | 1 | endless-terminals, tmax |
  51. | `gpqa-diamond` | Local seed-23 random GPQA Diamond subset | `terminus-2` | 100 | 1 | synthetic-data-kit |
  52. | `aime` | Full 30-task MathArena AIME 2026 dataset | `terminus-2` | 30 | 4 | synthetic-data-kit |
  53. If `TARGET_BENCHMARK` is omitted, `swe-bench-verified` is used.
  54. ### Evaluation Data Boundary
  55. `benchmarks/swe_bench_verified/runs/qwen35base_seed23_random100/` is read-only
  56. diagnostic context for the data-generation agent. It contains the raw
  57. Qwen3.5-35B-A3B-Base 100-task Harbor run, including per-task results,
  58. mini-swe-agent trajectories, verifier reports, and Tinker usage and cost logs.
  59. Agents may inspect these artifacts to understand benchmark distribution,
  60. failure modes, and runtime constraints. They must not copy, transform, filter,
  61. distill, or otherwise derive training records from benchmark tasks or
  62. artifacts. Training data must be newly synthesized from non-benchmark public
  63. sources, rather than invented without source grounding. Because this base run
  64. uses the same official SWE-bench Verified subset used for evaluation, every
  65. file under the directory must be treated as an evaluation-set artifact.
  66. ## Quick Start
  67. ### 1. Install Runtime Prerequisites
  68. - Python 3.12
  69. - A Tinker API key with access to `Qwen/Qwen3.5-35B-A3B-Base`
  70. - An E2B API key; the Pro plan is recommended for evaluation workloads
  71. - Claude Code CLI as `claude` when `DATA_AGENT_HARNESS=claude-code`
  72. - Codex CLI as `codex` when `DATA_AGENT_HARNESS=codex`
  73. - Kimi Code CLI as `kimi` when `DATA_AGENT_HARNESS=kimi-code`
  74. Install Claude Code with Anthropic's native installer on macOS, Linux, or WSL:
  75. ```bash
  76. curl -fsSL https://claude.ai/install.sh | bash
  77. claude --version
  78. ```
  79. Alternative official package-manager installs:
  80. ```bash
  81. # macOS
  82. brew install --cask claude-code
  83. # Windows
  84. winget install Anthropic.ClaudeCode
  85. ```
  86. Install Codex in a Node/npm environment:
  87. ```bash
  88. npm install -g @openai/codex
  89. codex --version
  90. ```
  91. Install Kimi Code with Moonshot's installer, or with npm on Node.js 22.19.0 or
  92. newer:
  93. ```bash
  94. curl -fsSL https://code.kimi.com/kimi-code/install.sh | bash
    • Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
    • Line 123 on GitHub
  95. kimi --version
  96. ```
  97. For interactive local Claude Code use, run `claude` once and complete its login
  98. flow. Automated RSIBench-Data sessions can instead use repository-local `.env`
  99. API configuration, described below.
  100. ### 2. Create the Python Environment
  101. ```bash
  102. bash tools/setup_env.sh
  103. source .venv/bin/activate
  104. ```
  105. ### 3. Initialize Seed Factories
  106. Seed factories are read-only references used by the data-generation agent. The
  107. selected benchmark profile exposes only the factories in its allowlist; a
  108. custom profile can opt into all factories with `"seed_factories": ["*"]`.
  109. ```bash
  110. git submodule update --init --recursive seed_factories
  111. ```
  112. The agent must inspect the exposed references when designing training data.
  113. ### 4. Configure Secrets and Models
  114. ```bash
  115. cp .env.example .env
  116. ```
  117. Required fields:
  118. ```dotenv
  119. TINKER_API_KEY=...
  120. E2B_API_KEY=...
  121. ```
  122. The model names and API-version strings below are **example values**, not a
  123. compatibility guarantee. Providers and account entitlements change over time:
  124. before starting a paid run, replace them with values supported by your current
  125. Tinker, Anthropic, and Codex-compatible endpoint accounts. The repository
  126. defaults `TARGET_MODEL` to `Qwen/Qwen3.5-35B-A3B-Base`; the documented session
  127. and base-model examples use that same model for consistency. Other
  128. Tinker-supported target models (for example a Qwen 3.6 variant, when available
  129. to your account) may be supplied explicitly as `TARGET_MODEL`.
  130. Common optional fields:
  131. ```dotenv
  132. # Claude Code harness only
  133. ANTHROPIC_BASE_URL=...
  134. ANTHROPIC_API_KEY=...
  135. ANTHROPIC_AUTH_TOKEN=
  136. ANTHROPIC_MODEL=claude-opus-4-8
  137. ANTHROPIC_DEFAULT_OPUS_MODEL=
  138. ANTHROPIC_DEFAULT_SONNET_MODEL=
  139. ANTHROPIC_DEFAULT_HAIKU_MODEL=
  140. ANTHROPIC_DEFAULT_FABLE_MODEL=
  141. CLAUDE_CODE_SUBAGENT_MODEL=
  142. ENABLE_TOOL_SEARCH=
  143. CLAUDE_CODE_AUTO_COMPACT_WINDOW=
  144. CLAUDE_CODE_EFFORT_LEVEL=
  145. CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=
  146. # Codex harness only
  147. CODEX_BASE_URL=...
  148. CODEX_API_KEY=...
  149. CODEX_MODEL=gpt-5.5
  150. CODEX_RESPONSES_API_VERSION=2025-04-01-preview
  151. # Generated rollout/API calls made by data-generation scripts
  152. DATA_AGENT_ROLL_BASE_URL=...
  153. DATA_AGENT_ROLL_API_KEY=...
  154. DATA_AGENT_ROLL_MODEL=claude-opus-4-8
  155. ```
  156. The Claude Code harness uses `ANTHROPIC_BASE_URL`, `ANTHROPIC_MODEL`, and either
  157. `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN`. `ANTHROPIC_MODEL` is also the
  158. default `DATA_AGENT_MODEL` for the Claude Code orchestrator. Additional Claude
  159. Code environment variables can be set in `.env` and are passed through to the
  160. Claude subprocess.
  161. Generated data scripts that perform rollout/API calls use
  162. `DATA_AGENT_ROLL_BASE_URL`, `DATA_AGENT_ROLL_API_KEY`, and
  163. `DATA_AGENT_ROLL_MODEL`.
  164. When `DATA_AGENT_HARNESS=claude-code`, the runner passes the configured Claude
  165. Code environment to the Claude subprocess. `ANTHROPIC_API_KEY` mode uses Claude
  166. Code's bare/project setting path; `ANTHROPIC_AUTH_TOKEN` mode is used as-is.
  167. Both credential fields may be left unset to use the caller's existing Claude CLI
  168. configuration.
  169. The Codex harness uses `CODEX_BASE_URL`, `CODEX_API_KEY`, and `CODEX_MODEL`.
  170. It creates a session-local `CODEX_HOME` and a custom model provider backed by
  171. the configured endpoint, so it does not depend on machine-level Codex login
  172. state. Codex sessions require both `CODEX_BASE_URL` and `CODEX_API_KEY`.
  173. The Kimi Code harness uses `KIMI_MODEL_NAME`, `KIMI_MODEL_API_KEY`, and optional
  174. `KIMI_MODEL_BASE_URL`. It creates a session-local `KIMI_CODE_HOME` with only the
  175. required permission setting, so it does not depend on machine-level Kimi login
  176. state. Kimi Code sessions use `default_permission_mode = "yolo"`.
  177. Runner scripts load `.env` in override mode. Repository-local values therefore
  178. take precedence over same-named variables in the caller's shell.
  179. ### Optional Docker Isolation
  180. The standard entrypoint still runs directly on the host:
  181. ```bash
  182. bash runner/run_session.sh
  183. ```
  184. To run the entire session inside Docker while mounting only this repository,
  185. build the runtime image and use the Docker wrapper:
  186. ```bash
  187. docker build --network=host \
  188. --build-arg CLAUDE_CODE_VERSION=2.1.197 \
  189. --build-arg CODEX_CLI_VERSION=0.142.5 \
  190. --build-arg HTTP_PROXY=http://127.0.0.1:7890 \
  191. --build-arg HTTPS_PROXY=http://127.0.0.1:7890 \
  192. -f docker/rsibench-runtime.Dockerfile \
  193. -t rsibench-runtime:ubuntu24-claude-2.1.197-codex-0.142.5 \
  194. docker
  195. bash tools/run_session_in_docker.sh
  196. ```
  197. The wrapper mounts `/data/RSIBench-Data` to the same path inside the container,
  198. loads `.env`, and then runs `bash runner/run_session.sh` in the container. This
  199. keeps the runner, data-agent CLI, Tinker calls, and official eval in one
  200. container view while preventing access to host directories that are not mounted,
  201. such as external archive folders. It also hides the repo-local
  202. `archived_artifacts/` directory with a container-only tmpfs mount, then archives
  203. the completed `RUN_ID` from `artifacts/runs/` into that host directory after the
  204. container exits. Keep `artifacts/` fresh before starting a new isolated run if
  205. old active run records should not be visible to the data agent.
  206. The image includes Claude Code `2.1.197` and Codex CLI `0.142.5` by default and
  207. reuses the repository `.venv` through the mount, so the container should use an
  208. Ubuntu 24.04-compatible host `.venv`. Override `RSIBENCH_DOCKER_IMAGE` to use a
  209. different runtime image.
  210. Omit the proxy build args when the host does not expose a local proxy on port
  211. `7890`.
  212. ### 5. Validate the Environment
  213. ```bash
  214. bash tools/check_environment.sh
  215. ```
  216. This check imports the real Tinker, E2B, and Harbor execution path and verifies
  217. that `TINKER_API_KEY` and `E2B_API_KEY` are present. If it fails, rerun
  218. `bash tools/setup_env.sh`, activate `.venv`, and check `.env`.
  219. ### 6. Run a One-Task Paid Smoke Test
  220. Use this before a budgeted session to exercise the actual Tinker sampling, E2B
  221. proxy, and Harbor path on one task. It does **not** test the Claude Code/Codex
  222. orchestration loop, and it can incur Tinker and E2B charges.
  223. ```bash
  224. set -a; source .env; set +a
  225. python3 -m backend.e2b_eval \
  226. --model-path Qwen/Qwen3.5-35B-A3B-Base \
  227. --run-dir artifacts/runs/smoke-base-qwen35 \
  228. --base-model Qwen/Qwen3.5-35B-A3B-Base \
  229. --n-tasks 1 --n-concurrent 1
  230. ```
  231. Success creates `artifacts/runs/smoke-base-qwen35/harbor/jobs/*/result.json`
  232. and `tinker_cost_estimate.json`. Inspect the result before increasing
  233. `--n-tasks` to 2 or 3 or starting a full session.
  234. ## Run an Experiment
  235. The main entrypoint is [`runner/run_session.sh`](runner/run_session.sh).
  236. ### Claude Code
  237. ```bash
  238. RUN_ID=qwen35base-claude-code-opus48-001 \
  239. DATA_AGENT_HARNESS=claude-code \
  240. TARGET_MODEL=Qwen/Qwen3.5-35B-A3B-Base \
  241. TARGET_BENCHMARK=swe-bench-verified \
  242. WALL_TIME_BUDGET_SEC=14400 \
  243. TINKER_COST_BUDGET_USD=25 \
  244. RUN_OFFICIAL_EVAL=1 \
  245. bash runner/run_session.sh
  246. ```
  247. ### Claude Code with Kimi
  248. For Kimi runs, put the Claude Code provider settings in `.env` and run the
  249. Claude Code harness normally.
  250. ```dotenv
  251. ANTHROPIC_BASE_URL=https://api.moonshot.cn/anthropic
  252. ANTHROPIC_AUTH_TOKEN=<YOUR_MOONSHOT_API_KEY>
  253. ANTHROPIC_MODEL=kimi-k3[1m]
  254. ANTHROPIC_DEFAULT_OPUS_MODEL=kimi-k3[1m]
  255. ANTHROPIC_DEFAULT_SONNET_MODEL=kimi-k3[1m]
  256. ANTHROPIC_DEFAULT_HAIKU_MODEL=kimi-k3[1m]
  257. ANTHROPIC_DEFAULT_FABLE_MODEL=kimi-k3[1m]
  258. CLAUDE_CODE_SUBAGENT_MODEL=kimi-k3[1m]
  259. ENABLE_TOOL_SEARCH=false
  260. CLAUDE_CODE_AUTO_COMPACT_WINDOW=1048576
  261. CLAUDE_CODE_EFFORT_LEVEL=max
  262. ```
  263. ```bash
  264. RUN_ID=qwen35base-kimi-k3-claude-code-001 \
  265. DATA_AGENT_HARNESS=claude-code \
  266. TARGET_MODEL=Qwen/Qwen3.5-35B-A3B-Base \
  267. TARGET_BENCHMARK=swe-bench-verified \
  268. WALL_TIME_BUDGET_SEC=14400 \
  269. TINKER_COST_BUDGET_USD=25 \
  270. RUN_OFFICIAL_EVAL=1 \
  271. bash runner/run_session.sh
  272. ```
  273. ### Codex
  274. Claude Code and Codex receive the same generated workspace prompt and
  275. instruction files. Codex-specific code changes only how that prompt is
  276. executed.
  277. ```bash
  278. RUN_ID=qwen35base-codex-gpt55-001 \
  279. DATA_AGENT_HARNESS=codex \
  280. TARGET_MODEL=Qwen/Qwen3.5-35B-A3B-Base \
  281. TARGET_BENCHMARK=swe-bench-verified \
  282. WALL_TIME_BUDGET_SEC=14400 \
  283. TINKER_COST_BUDGET_USD=25 \
  284. RUN_OFFICIAL_EVAL=1 \
  285. bash runner/run_session.sh
  286. ```
  287. ### Kimi Code
  288. ```dotenv
  289. KIMI_MODEL_NAME=kimi-k3[1m]
  290. KIMI_MODEL_API_KEY=<YOUR_MOONSHOT_API_KEY>
  291. # KIMI_MODEL_BASE_URL=https://api.moonshot.ai/v1
  292. # KIMI_MODEL_THINKING_EFFORT=max
  293. ```
  294. ```bash
  295. RUN_ID=qwen35base-kimi-code-k3-001 \
  296. DATA_AGENT_HARNESS=kimi-code \
  297. TARGET_MODEL=Qwen/Qwen3.5-35B-A3B-Base \
  298. TARGET_BENCHMARK=swe-bench-verified \
  299. WALL_TIME_BUDGET_SEC=14400 \
  300. TINKER_COST_BUDGET_USD=25 \
  301. RUN_OFFICIAL_EVAL=1 \
  302. bash runner/run_session.sh
  303. ```
  304. For Claude Code, `DATA_AGENT_MODEL` defaults to `ANTHROPIC_MODEL`. For Codex,
  305. it defaults to `CODEX_MODEL`. For Kimi Code, it defaults to `KIMI_MODEL_NAME`.
  306. Generated scripts must use the separate `DATA_AGENT_ROLL_BASE_URL`,
  307. `DATA_AGENT_ROLL_API_KEY`, and `DATA_AGENT_ROLL_MODEL` values for direct
  308. rollout/API calls.
  309. ### Confirm a Successful Session
  310. With `RUN_OFFICIAL_EVAL=1`, a successful command exits with code 0 and prints
  311. these key log lines (with run-specific paths):
  312. ```text
  313. final_selection=.../agent_workspace/final_selection.json
  314. running official eval for final selection
  315. .../official_eval/official_eval.json
  316. ```
  317. The required final artifacts are:
  318. - `artifacts/runs/<RUN_ID>/agent_workspace/final_selection.json` — a completed
  319. attempt was selected.
  320. - `artifacts/runs/<RUN_ID>/official_eval/official_eval.json` — its `status` is
  321. `"completed"` and `return_code` is `0`.
  322. - `artifacts/runs/<RUN_ID>/official_eval/harbor/jobs/*/result.json` — the
  323. Harbor result used by the official evaluation.
  324. For a concise status check, run:
  325. ```bash
  326. python3 - <<'PY'
  327. import json
  328. from pathlib import Path
  329. run_id = "replace-with-your-run-id"
  330. root = Path("artifacts/runs") / run_id
  331. for path in (
  332. root / "agent_workspace/final_selection.json",
  333. root / "official_eval/official_eval.json",
  334. ):
  335. print(path, json.loads(path.read_text()))
  336. PY
  337. ```
  338. `RUN_OFFICIAL_EVAL=0` intentionally skips the final evaluation, so a
  339. `final_selection.json` alone means only that a checkpoint was selected; it is
  340. not a completed official run.
  341. ### Additional Benchmark Preparation
  342. The SWE-bench Multilingual profile uses a fixed local seed-23 random 100-task
  343. subset generated from `SWE-bench/SWE-bench_Multilingual`:
  344. ```bash
  345. python tools/create_swe_multilingual_local_dataset.py \
  346. --seed 23 \
  347. --n-tasks 100 \
  348. --output-dir benchmarks/swe_bench_multilingual/datasets/seed23_random_100 \
  349. --overwrite
  350. ```
  351. Set `TARGET_BENCHMARK=swe-bench-multilingual` to use that subset.
  352. For GPQA Diamond, set `TARGET_BENCHMARK=gpqa-diamond`. The profile uses
  353. `benchmarks/gpqa_diamond/datasets/seed23_random_100` with `terminus-2` and
  354. exposes Meta Llama's `seed_factories/synthetic-data-kit`.
  355. For AIME, set `TARGET_BENCHMARK=aime`. The profile uses
  356. `benchmarks/aime/datasets/full_30_2026` with `terminus-2`, evaluates each task
  357. with four attempts, and also exposes `seed_factories/synthetic-data-kit`.
  358. ## Session Contract
  359. The runner starts a PostTrainBench-style, non-interactive agent session.
  360. `DATA_AGENT_REPROMPT=1` is enabled by default. If Claude Code exits before
  361. calling `final_submit.sh <attempt_id>`, the runner invokes `claude --continue`
  362. while wall time and Tinker budget remain. Set `DATA_AGENT_REPROMPT=0` only when
  363. a single Claude Code turn is intentional.
  364. The generated agent workspace exposes four commands:
  365. ```text
  366. timer.sh
  367. budget_status.sh
  368. run_attempt.sh <attempt_id>
  369. final_submit.sh <attempt_id>
  370. ```
  371. - `timer.sh` reports the remaining wall-clock budget.
  372. - `budget_status.sh` reports committed attempt cost plus the estimate for any
  373. active attempt.
  374. - `run_attempt.sh` validates, trains, and evaluates one candidate dataset.
  375. - `final_submit.sh` selects the checkpoint that official final evaluation will
  376. reuse.
  377. ### Attempt Inputs
  378. Each attempt must contain:
  379. ```text
  380. attempts/<attempt_id>/train_messages.jsonl
  381. attempts/<attempt_id>/run_config.json # optional
  382. ```
  383. `train_messages.jsonl` is JSONL. Every line is a JSON object with a `messages`
  384. list containing at least one user message and one trainable assistant message.
  385. Assistant messages are trainable unless `"loss": false` is set.
  386. Before real Tinker training starts, `run_attempt.sh` executes the same runtime
  387. check as `tools/check_environment.sh`. Missing packages or API keys therefore
  388. fail before training budget is spent. It then writes:
  389. ```text
  390. attempts/<attempt_id>/tinker_preflight_estimate.json
  391. ```
  392. The preflight file estimates scheduled training tokens and training cost from
  393. `train_messages.jsonl` and `run_config.json`. Evaluation sampling cost is
  394. measured during Harbor evaluation from actual prompt and output tokens.
  395. ### Run Configuration
  396. Supported `run_config.json` keys:
  397. ```json
  398. {
  399. "train_max_steps": 1,
  400. "train_max_length": 65536,
  401. "train_learning_rate": "1e-4",
  402. "train_batch_size": 1,
  403. "train_gradient_accumulation_steps": 1,
  404. "train_weight_decay": 0.0,
  405. "train_grad_clip_norm": 0.0,
  406. "train_lora_rank": 8,
  407. "train_seed": 0,
  408. "train_max_records": 0,
  409. "train_on_all_assistant": false,
  410. "tinker_proxy_max_context_tokens": 65536,
  411. "tinker_proxy_max_prompt_tokens": 61439,
  412. "tinker_proxy_max_tokens": 4096,
  413. "tinker_temperature": 1.0,
  414. "tinker_top_p": 0.95,
  415. "tinker_top_k": 20,
  416. "eval_step_limit": 200,
  417. "n_concurrent": 32,
  418. "n_tasks": 100,
  419. "eval_timeout_sec": 0
  420. }
  421. ```
  422. Unknown keys are rejected. Proxy context and decoding settings, along with the
  423. evaluation step limit, are fixed to match official evaluation; the validator
  424. rejects attempts that change those evaluation hyperparameters. The legacy key
  425. `mini_swe_step_limit` remains accepted as an alias for older SWE runs.
  426. Every attempt evaluates the same official setting. When final evaluation is
  427. enabled, the runner reuses only the selected Tinker sampler checkpoint and
  428. reruns that setting. Harbor task errors and timeouts are recorded as
  429. `harbor_errors` and reflected in the score, but a scored Harbor result remains
  430. a completed, submittable attempt. The agent must call
  431. `final_submit.sh <attempt_id>` to select its final checkpoint.
  432. ### Process and Deadline Safety
  433. Attempts must run in the foreground:
  434. ```bash
  435. bash run_attempt.sh <attempt_id>
  436. ```
  437. Do not launch attempts with `nohup`, `&`, `disown`, `setsid`, watcher scripts,
  438. or task/waiter mechanisms. If a continuation finds an attempt already running,
  439. it must wait for that process rather than launch a duplicate.
  440. The agent must also protect final submission near the deadline:
  441. 1. Before starting a new attempt after a completed attempt exists, compare the
  442. remaining wall time with the last full attempt runtime plus a 30-minute
  443. submission buffer.
  444. 2. If the next attempt increases training steps, records, LoRA rank, or maximum
  445. length, assume it needs at least two hours unless a measurement shows
  446. otherwise.
  447. 3. During the final two hours, do not start a heavier or speculative attempt
  448. over a completed best attempt; submit the best completed attempt instead.
  449. 4. With less than one hour remaining, immediately run
  450. `final_submit.sh <best_attempt_id>` when any completed attempt exists.
  451. The benchmark profile fixes the official evaluation shape. Attempts always use
  452. the selected profile's `OFFICIAL_EVAL_N_TASKS`, `OFFICIAL_EVAL_N_CONCURRENT`,
  453. and `OFFICIAL_EVAL_STEP_LIMIT`. SWE-bench Verified evaluates the first 100 tasks
  454. in the configured Harbor dataset order by default; Terminal-Bench 2.0 evaluates
  455. 89 tasks. Official subsets and public base-run artifacts are analysis-only and
  456. cannot be used as training-data sources.
  457. ## Evaluation Runtime
  458. Each attempt creates and tears down its own E2B sandboxes: one tool-call proxy
  459. sandbox and the per-task Harbor sandboxes. The proxy selects its Tinker backend
  460. automatically:
  461. - Saved `tinker://.../sampler_weights/...` checkpoints use the Tinker SDK
  462. `SamplingClient`.
  463. - Base model names such as `Qwen/Qwen3.5-35B-A3B-Base` also use the Tinker SDK
  464. `SamplingClient`.
  465. The proxy uses the Tinker SDK `SamplingClient` by default for both saved
  466. `tinker://.../sampler_weights/...` checkpoints and base model names such as
  467. `Qwen/Qwen3.5-35B-A3B-Base`. The first SDK-backed evaluation builds the fixed
  468. E2B template `rsibench-data-tinker-proxy`; subsequent runs reuse it.
  469. ### Direct Checkpoint Evaluation
  470. To evaluate a trained checkpoint outside a budgeted session:
  471. ```bash
  472. set -a; source .env; set +a # TINKER_API_KEY + E2B_API_KEY
  473. python3 -m backend.e2b_eval \
  474. --model-path 'tinker://.../sampler_weights/...' \
  475. --run-dir artifacts/runs/manual-eval \
  476. --base-model Qwen/Qwen3.5-35B-A3B-Base \
  477. --harbor-dataset swe-bench/swe-bench-verified \
  478. --harbor-agent mini-swe-agent \
  479. --agent-config-profile mini_swe_agent \
  480. --n-tasks 100 --n-concurrent 32 --step-limit 200
  481. ```
  482. Use the following substitutions for other profiles:
  483. | Profile | Dataset | Agent | Additional arguments |
  484. |---|---|---|---|
  485. | Terminal-Bench 2.0 | `terminal-bench/terminal-bench-2` | `terminus-2` | `--agent-config-profile terminus_2 --n-tasks 89` |
  486. | GPQA Diamond | `benchmarks/gpqa_diamond/datasets/seed23_random_100` | `terminus-2` | `--agent-config-profile terminus_2 --n-tasks 100` |
  487. | AIME | `benchmarks/aime/datasets/full_30_2026` | `terminus-2` | `--agent-config-profile terminus_2 --n-tasks 30 --n-attempts 4` |
  488. ### Direct Base-Model Evaluation
  489. Pass a supported base model name as `--model-path` to evaluate it directly:
  490. ```bash
  491. set -a; source .env; set +a
  492. python3 -m backend.e2b_eval \
  493. --model-path Qwen/Qwen3.5-35B-A3B-Base \
  494. --run-dir artifacts/runs/manual-base-eval \
  495. --base-model Qwen/Qwen3.5-35B-A3B-Base \
  496. --n-tasks 50 --n-concurrent 32
  497. ```
  498. Direct evaluation starts the proxy sandbox, runs Harbor on E2B, and writes
  499. `harbor/jobs/*/result.json` plus `tinker_cost_estimate.json` under the run
  500. directory. Sampling rates default to `$0.36/M` prefill tokens and `$0.89/M`
  501. sample tokens. Override them with `TINKER_PROXY_PREFILL_USD_PER_1M` and
  502. `TINKER_PROXY_SAMPLE_USD_PER_1M`.
  503. ## Artifacts and Accounting
  504. Session outputs are written under `artifacts/runs/<run_id>/`.
  505. | Artifact | Purpose |
  506. |---|---|
  507. | `agent_initial.jsonl` | Initial Claude Code stdout and stream events |
  508. | `agent_continue_<n>.jsonl` | Continuation stream events when reprompting is enabled |
  509. | `agent_custom_command.log` | Output from custom harness commands |
  510. | `data_agent_usage.json` | Orchestrator usage and reported rollout/API usage |
  511. | `data_agent_api_usage.jsonl` | Per-response usage from generated rollout/API calls |
  512. | `harbor/jobs/*/result.json` | Harbor task and aggregate evaluation results |
  513. | Attempt Harbor trajectories | Evaluated agent behavior and verifier evidence |
  514. Generated data scripts that perform rollout/API calls should use
  515. `runner.lib.claude_usage_client.create_message`. The wrapper uses only the
  516. `DATA_AGENT_ROLL_BASE_URL`, `DATA_AGENT_ROLL_API_KEY`, and
  517. `DATA_AGENT_ROLL_MODEL` configuration and appends one record per response to
  518. `$DATA_AGENT_API_USAGE_LOG`, including `model`, `input_tokens`,
  519. `cache_creation_input_tokens`, `cache_read_input_tokens`, and `output_tokens`.
  520. This usage is retained for experiment accounting and does not consume the
  521. Tinker budget.
  522. ## Failure Recovery
  523. - **E2B timeout or proxy failure:** retain the run directory and inspect the
  524. attempt's `harbor_errors` and Harbor `result.json`. Confirm the E2B key,
  525. account capacity, and transient service status, then run the one-task smoke
  526. test again. Do not treat an infrastructure error as a score or repeatedly
  527. spend budget on the same failure.
  528. - **Tinker training failure:** inspect
  529. `artifacts/runs/<RUN_ID>/agent_workspace/attempts/<attempt_id>/tinker_preflight_estimate.json`
  530. and the matching `artifacts/runs/<RUN_ID>-<attempt_id>/` logs. Correct the
  531. dataset/configuration issue or account access, then create a new attempt;
  532. failed attempts cannot be selected by `final_submit.sh`.
  533. - **Budget exhausted:** no further attempt can start. If a completed attempt
  534. was already selected, run `RUN_ID=<RUN_ID> bash runner/run_official_eval.sh`
  535. to retry only the final evaluation after fixing infrastructure access. If no
  536. `final_selection.json` exists, the session did not complete and needs a new
  537. budgeted run.
  538. - **Claude Code or Codex exits early:** `DATA_AGENT_REPROMPT=1` (the default)
  539. automatically continues the harness while wall time and Tinker budget remain.
  540. If the runner exits without `final_selection.json`, inspect
  541. `agent_initial.jsonl`, `agent_continue_*.jsonl`, and `budget_status.sh`.
  542. Re-running with the same `RUN_ID` preserves accounting but does not pause its
  543. original wall-clock deadline; use it only while time remains.
  544. ## Development Checks
  545. Run the shell and unit checks before pushing changes:
  546. ```bash
  547. bash -n \
  548. runner/run_session.sh \
  549. runner/run_official_eval.sh \
  550. runner/workspace_template/pipeline/run_chain.sh \
  551. runner/workspace_template/pipeline/train.sh \
  552. tools/check_environment.sh \
  553. tools/setup_env.sh
  554. PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests
  555. ```
  556. ## Repository Layout
  557. ```text
  558. backend/e2b_eval/ E2B proxy and Harbor evaluation runtime
  559. backend/tinker/ Tinker training, pricing, and cost estimates
  560. benchmarks/ Benchmark profiles, datasets, and context
  561. docs/tinker_train_eval.md Tinker training and evaluation runbook
  562. docs/tinker_cookbook.md Local Tinker API notes for data agents
  563. runner/ Budgeted session and official-eval entrypoints
  564. runner/lib/ Validation, accounting, budget, and audit helpers
  565. runner/workspace_template/pipeline/ Fixed training and E2B evaluation chain
  566. seed_factories/ Read-only synthetic-data reference projects
  567. skills/rsibench-data-factory/ Data-factory operating instructions
  568. tests/ Unit tests for runner and evaluation contracts
  569. tools/ Environment and dataset preparation utilities
  570. ```
  571. For deeper implementation details, see
  572. [`docs/tinker_train_eval.md`](docs/tinker_train_eval.md),
  573. [`docs/tinker_cookbook.md`](docs/tinker_cookbook.md), and the
  574. [`RSIBench-Data factory skill`](skills/rsibench-data-factory/SKILL.md).
  575. [website-image]: https://img.shields.io/badge/Website-rsibench.co-0f141b
  576. [website-url]: http://rsibench.co
  577. [evolvent-image]: https://img.shields.io/badge/Evolvent_AI-evolvent.co-0f141b
  578. [evolvent-url]: https://evolvent.co
  579. [x-image]: https://img.shields.io/twitter/follow/Evolvent_AI?style=social
  580. [x-url]: https://x.com/Evolvent_AI
  581. [linkedin-image]: https://img.shields.io/badge/LinkedIn-Evolvent_AI-0A66C2?logo=linkedin&logoColor=white
  582. [linkedin-url]: https://www.linkedin.com/company/evolvent-ai
  583. [license-image]: https://img.shields.io/badge/License-CC_BY--NC_4.0-blue.svg
  584. [license-url]: https://creativecommons.org/licenses/by-nc/4.0/

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.