Skip to content
Back to search

SAfactory

SAfactory: A Scalable Agentic Infrastructure for Training Trustworthy Autonomous Intelligence

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/AI45Lab/SAfactory with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/AI45Lab/SAfactory'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/ai45lab--safactory/badge.svg)](/listings/ai45lab--safactory)

HTML

<a href="/listings/ai45lab--safactory"><img src="/api/listings/ai45lab--safactory/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

1. SAfactory Installation And Gateway Configuration

Before running Geo3K, prepare the runtime image and dataset as described in [Standard Environment: Geo3K](docs/reference/environments.md#standard-environment-geo3k).

Install SAfactory:

git clone https://github.com/AI45Lab/SAfactory.git
cd SAfactory
pip install -U -r requirements.txt

Docker must be available locally, and the current user must be able to run `docker build`, `docker run`, and `docker exec`.

Create a local Gateway config:

cp gateway/config.example.yaml gateway/config.local.yaml

Edit `gateway/config.local.yaml` and explicitly set the storage path and model route:

listen_host: 0.0.0.0
listen_port: 8000
base_session_path: /v1/sessions
max_steps: -1

storage_type: sqlite
storage_config:
  db_url: sqlite://env_trajs.db

llm_routes:
  geo3k_model:
    base_url: http://YOUR_LLM_HOST/v1
    api_key: YOUR_API_KEY
    supports_stream: true
    max_concurrency: 64

Start the Gateway:

python -m gateway --config gateway/config.local.yaml

Check readiness from another terminal:

curl http://127.0.0.1:8000/readyz

When using Cloud storage, the recommended configuration is to omit `landing_table` and `env_config_table` from the Gateway YAML and set only one profile in `.env`:

WT_SDK_PROFILE=test

or:

WT_SDK_PROFILE=production

The `test` profile selects `v2_landing_test` and `env_config_test`; the `production`/`prod` profile selects `wind_tunnel_landing` and `evaluation_env_config`. This keeps trajectory and environment-config writes in the same environment. Explicit table names in `storage_config` override the profile and should be used only for intentional special cases. SAfactory does not access the serving table.

2. Minimal Evaluation With Geo3K

Run a minimal Geo3K evaluation in Docker mode:

python launcher.py \
  --mode docker \
  --agent-config env/geo3k/geo3k_config.yaml \
  --agent-start-config env/geo3k/geo3k_start.yaml \
  --gateway-base-url http://127.0.0.1:8000/v1/sessions \
  --llm-model geo3k_model \
  --enable-evaluation \
  --db-path sqlite://env_trajs.db \
  --job-id geo3k-docker-smoke \
  --pool-size 1 \
  --max-workers 1 \
  --max-steps 10

`--llm-model` must match a key under `llm_routes`. With `--enable-evaluation`, SAfactory calls `env/geo3k/rule_evaluator.py` and writes the final reward.

3. Minimal Training With Geo3K

Geo3K training uses the RL bridge under `rl/` and the example config at `rl/examples/geo3k_vl/env.sh`.

Before starting, edit `rl/examples/geo3k_vl/env.sh`, or export the same variables in each terminal. Set the required local paths and scale the first run down:

export AIEVOBOX_ROOT=$(pwd)
export AIEVOBOX_MODE=docker
export AIEVOBOX_AGENT_CONFIG=${AIEVOBOX_ROOT}/env/geo3k/geo3k_config.yaml
export AIEVOBOX_AGENT_START_CONFIG=${AIEVOBOX_ROOT}/env/geo3k/geo3k_start.yaml
export AIEVOBOX_GATEWAY_HOST=127.0.0.1
export AIEVOBOX_GATEWAY_PORT=8000
export RL_MODEL=geo3k_model
export AIEVOBOX_POOL_SIZE=2
export RL_GROUP_SIZE=2
export RL_EPOCH=1
export HF_CKPT_DIR=/path/to/hf-checkpoint
export SLIME_HOME=/path/to/slime
export MEGATRON_HOME=/path/to/Megatron-LM

Then start two processes from the repository root:

RL_ENV_SH=rl/examples/geo3k_vl/env.sh bash rl/run_slime_generator.sh
RL_ENV_SH=rl/examples/geo3k_vl/env.sh bash rl/run_buffer_server.sh

Rewards for RL training come from completed trajectories in the database: `rule_evaluator.py` writes the reward after evaluation, and Buffer Server fetches trajectory rows with rewards from the same database used by Launcher / Gateway before serving them to the training process.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 278 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <div align="center">
  2. # SAfactory
  3. <p align="center">
  4. <a href="README_CN.md">中文</a> &nbsp;|&nbsp; English
  5. </p>
  6. **SAfactory is a scalable infrastructure for agent evaluation, trajectory collection, and reinforcement learning training. It schedules agents and benchmark environments as external runtimes, routes model calls through a session-aware OpenAI-compatible Gateway, records trajectories, and feeds completed rollouts into training systems such as Slime.**
  7. <p align="center">
  8. <a href="#why-safactory">Why SAfactory</a> •
  9. <a href="#demo">Demo</a> •
  10. <a href="#agent-skill">Agent Skill</a> •
  11. <a href="#quick-start">Quick Start</a> •
  12. <a href="#documentation">Documentation</a> •
  13. <a href="#citation">Citation</a>
  14. </p>
  15. ![Python](https://img.shields.io/badge/python-3.9%2B-blue)
  16. ![License](https://img.shields.io/badge/license-MIT-green)
  17. ![Execution](https://img.shields.io/badge/mode-docker%20%7C%20rjob%20%7C%20sandbox-orange)
  18. ![LLM](https://img.shields.io/badge/LLM-OpenAI--compatible-purple)
  19. </div>
  20. ---
  21. ## <a id="why-safactory"></a>✨ Why SAfactory
  22. SAfactory provides one unified workflow for agent onboarding, benchmark onboarding, evaluation, rollout data generation, and RL training.
  23. The core runtime contract is:
  24. - one dataset row becomes one scheduled episode;
  25. - each episode gets its own `session_id` and Gateway session;
  26. - the runtime calls the target model through the Gateway;
  27. - the runtime returns one JSON result;
  28. - `rule_evaluator.py` converts runtime output and trajectory data into the SAfactory format;
  29. - completed trainable trajectories can be consumed by the RL Buffer Server or persisted as reusable data assets.
  30. ## <a id="demo"></a>🎬 Demo
  31. <div align="center">
  32. https://github.com/user-attachments/assets/4c551b27-ce4d-4fc8-8df6-d6dc8100cc88
  33. *Click to watch the full demo*
  34. </div>
  35. ## <a id="agent-skill"></a>🧩 Agent Skill Quick Start
  36. This repository includes an Agent skill for template-based environment integration, local contract tests, optional Docker/RJob evaluation, and GRPO/RL workflows:
  37. ```text
  38. skills/safactory-workflows/SKILL.md
  39. ```
  40. ### <a id="benchmark-onboarding-prompt"></a>Benchmark Onboarding Prompt
  41. Provide the benchmark source, one-row dataset schema, 1–2 representative cases, native single-case command, and native output format. Specify Docker or RJob deployment when known. **Integration alone does not require evaluation, a score/reward definition, a running Gateway, or an internal cluster.** Only provide scoring details when evaluation is requested.
  42. The Agent copies fixed runner/config templates, fills the environment hooks, and runs local contract tests with an owned mock model endpoint. Native dependencies must be available locally or represented by explicit test fixtures. Live deployment is a separate check when image/data/model/runtime access is available; a helper starts and stops Gateway without a second terminal. RJob live runs additionally require an existing cluster configuration and a cluster-reachable Gateway URL.
  43. <details>
  44. <summary>Expand to get the Benchmark Onboarding Prompt</summary>
  45. ```text
  46. Use skills/safactory-workflows to onboard this benchmark into SAfactory.
  47. [Goal]
  48. - evaluation: [disabled (default, integration only) / enabled]
  49. - target deployment mode: [docker / rjob / decide after adapter inspection]
  50. - validation: [local contract first / also run live deployment when available]
  51. [Benchmark]
  52. - environment name: ____________________
  53. - source or checkout path/repository: ____________________
  54. - dataset path and one-row schema: ____________________
  55. - 1–2 case IDs/rows: ____________________
  56. - native single-case command, or README file/section: ____________________
  57. - Docker image, if available: ____________________
  58. - native result/output path and format: ____________________
  59. [Only if evaluation is enabled]
  60. - native score field/file, range, meaning, and pass condition: ____________________
  61. Start from the fixed templates based on docs/guides/custom-environment.md.
  62. Keep protocol handling in runner.py and fill adapter.py with single-case row
  63. mapping, Gateway model routing, native command execution, and output collection.
  64. Do not reimplement native benchmark solving/scoring logic.
  65. Only add rule_evaluator.py and --enable-evaluation if evaluation is enabled.
  66. Run local contract checks without requiring Gateway or RJob cluster setup.
  67. Report fixtures used, native output mapping, and which checks actually passed;
  68. keep local contract, live deployment, and evaluation results distinct.
  69. Ask only for missing information needed by the next dependent step.
  70. ```
  71. </details>
  72. See [Custom Environments](docs/guides/custom-environment.md) and the skill's [integration reference](skills/safactory-workflows/references/environment-integration.md) for the templates and Docker/RJob deployment settings. Verify any environment with one command — `python skills/safactory-workflows/scripts/check_environment.py --env env/<name>` — which runs static consistency checks, a contract smoke test against an owned mock endpoint, and an optional live stage, labeling each failure with the side that owns it. If your Agent cannot discover local skills automatically, include `skills/safactory-workflows/` explicitly in the prompt.
  73. ## <a id="quick-start"></a>🚀 Quick Start
  74. ### 1. SAfactory Installation And Gateway Configuration
  75. Before running Geo3K, prepare the runtime image and dataset as described in [Standard Environment: Geo3K](docs/reference/environments.md#standard-environment-geo3k).
  76. Install SAfactory:
  77. ```bash
  78. git clone https://github.com/AI45Lab/SAfactory.git
  79. cd SAfactory
  80. pip install -U -r requirements.txt
  81. ```
  82. Docker must be available locally, and the current user must be able to run `docker build`, `docker run`, and `docker exec`.
  83. Create a local Gateway config:
  84. ```bash
  85. cp gateway/config.example.yaml gateway/config.local.yaml
  86. ```
  87. Edit `gateway/config.local.yaml` and explicitly set the storage path and model route:
  88. ```yaml
  89. listen_host: 0.0.0.0
  90. listen_port: 8000
  91. base_session_path: /v1/sessions
  92. max_steps: -1
  93. storage_type: sqlite
  94. storage_config:
  95. db_url: sqlite://env_trajs.db
  96. llm_routes:
  97. geo3k_model:
  98. base_url: http://YOUR_LLM_HOST/v1
  99. api_key: YOUR_API_KEY
  100. supports_stream: true
  101. max_concurrency: 64
  102. ```
  103. Start the Gateway:
  104. ```bash
  105. python -m gateway --config gateway/config.local.yaml
  106. ```
  107. Check readiness from another terminal:
  108. ```bash
  109. curl http://127.0.0.1:8000/readyz
  110. ```
  111. When using Cloud storage, the recommended configuration is to omit
  112. `landing_table` and `env_config_table` from the Gateway YAML and set only one
  113. profile in `.env`:
  114. ```bash
  115. WT_SDK_PROFILE=test
  116. ```
  117. or:
  118. ```bash
  119. WT_SDK_PROFILE=production
  120. ```
  121. The `test` profile selects `v2_landing_test` and `env_config_test`; the
  122. `production`/`prod` profile selects `wind_tunnel_landing` and
  123. `evaluation_env_config`. This keeps trajectory and environment-config writes
  124. in the same environment. Explicit table names in `storage_config` override the
  125. profile and should be used only for intentional special cases. SAfactory does
  126. not access the serving table.
  127. ### 2. Minimal Evaluation With Geo3K
  128. Run a minimal Geo3K evaluation in Docker mode:
  129. ```bash
  130. python launcher.py \
  131. --mode docker \
  132. --agent-config env/geo3k/geo3k_config.yaml \
  133. --agent-start-config env/geo3k/geo3k_start.yaml \
  134. --gateway-base-url http://127.0.0.1:8000/v1/sessions \
  135. --llm-model geo3k_model \
  136. --enable-evaluation \
  137. --db-path sqlite://env_trajs.db \
  138. --job-id geo3k-docker-smoke \
  139. --pool-size 1 \
  140. --max-workers 1 \
  141. --max-steps 10
  142. ```
  143. `--llm-model` must match a key under `llm_routes`. With `--enable-evaluation`, SAfactory calls `env/geo3k/rule_evaluator.py` and writes the final reward.
  144. ### 3. Minimal Training With Geo3K
  145. Geo3K training uses the RL bridge under `rl/` and the example config at `rl/examples/geo3k_vl/env.sh`.
  146. Before starting, edit `rl/examples/geo3k_vl/env.sh`, or export the same variables in each terminal. Set the required local paths and scale the first run down:
  147. ```bash
  148. export AIEVOBOX_ROOT=$(pwd)
  149. export AIEVOBOX_MODE=docker
  150. export AIEVOBOX_AGENT_CONFIG=${AIEVOBOX_ROOT}/env/geo3k/geo3k_config.yaml
  151. export AIEVOBOX_AGENT_START_CONFIG=${AIEVOBOX_ROOT}/env/geo3k/geo3k_start.yaml
  152. export AIEVOBOX_GATEWAY_HOST=127.0.0.1
  153. export AIEVOBOX_GATEWAY_PORT=8000
  154. export RL_MODEL=geo3k_model
  155. export AIEVOBOX_POOL_SIZE=2
  156. export RL_GROUP_SIZE=2
  157. export RL_EPOCH=1
  158. export HF_CKPT_DIR=/path/to/hf-checkpoint
  159. export SLIME_HOME=/path/to/slime
  160. export MEGATRON_HOME=/path/to/Megatron-LM
  161. ```
  162. Then start two processes from the repository root:
  163. ```bash
  164. RL_ENV_SH=rl/examples/geo3k_vl/env.sh bash rl/run_slime_generator.sh
  165. ```
  166. ```bash
  167. RL_ENV_SH=rl/examples/geo3k_vl/env.sh bash rl/run_buffer_server.sh
  168. ```
  169. Rewards for RL training come from completed trajectories in the database: `rule_evaluator.py` writes the reward after evaluation, and Buffer Server fetches trajectory rows with rewards from the same database used by Launcher / Gateway before serving them to the training process.
  170. Buffer Server can automatically start a Gateway and route `RL_MODEL` to the Slime-hosted LLM proxy. If a manually started Gateway is already using the same port, stop it first. Set `AIEVOBOX_GATEWAY_AUTOSTART=0` only when the external Gateway already has the correct route and storage configuration.
  171. ## <a id="documentation"></a>📚 Documentation
  172. ### Guides
  173. | Guide | What it covers |
  174. |-------|----------------|
  175. | [Custom Environments](docs/guides/custom-environment.md) | How to onboard a new external runtime adapter. |
  176. | [Evaluation](docs/guides/evaluation.md) | Rule evaluator discovery, interfaces, reward writing, and Geo3K evaluation. |
  177. | [RL Training](docs/guides/rl-training.md) | Buffer Server, Slime generator, Geo3K training path, and key variables. |
  178. | [Data Manager](docs/guides/data-manager.md) | SQLite storage behavior, table schema, row types, and query examples. |
  179. | [S3 + LanceDB Storage](docs/guides/S3+LanceDB-storage.md) | How to switch trajectory storage from local SQLite to S3 + LanceDB. |
  180. ### Internal
  181. | Internal Doc | What it covers |
  182. |--------------|----------------|
  183. | [RJob Mode](docs/internal/rjob-mode.md) | Remote RJob runtime config, authentication, mounts, Gateway reachability, and Geo3K examples. |
  184. | [Sandbox Mode](docs/internal/sandbox-mode.md) | Brainbox Sandbox Environment config, volumes, lifecycle, and launch flow. |
  185. ### Reference
  186. | Reference | What it covers |
  187. |-----------|----------------|
  188. | [CLI And Configuration](docs/reference/configuration.md) | Launcher flags, Gateway config, agent config, start config, RJob, and Sandbox settings. |
  189. | [Supported Environments](docs/reference/environments.md) | Checked-in adapters, the standard Geo3K path, and the runtime matrix. |
  190. | [Gateway Reference](docs/reference/gateway.md) | OpenAI-compatible routes, session endpoints, telemetry, request logs, and storage consistency. |
  191. | [Report](https://arxiv.org/pdf/2605.06230) | SAfactory report. |
  192. ## <a id="citation"></a>📖 Citation
  193. If SAfactory or SAfactory-generated datasets are useful in your work, please cite this repository and the specific dataset or report you used.
  194. ```bibtex
  195. @misc{chen2026safactoryscalableagenticinfrastructure,
  196. title={SAfactory: A Scalable Agentic Infrastructure for Training Trustworthy Autonomous Intelligence},
  197. author={Shanghai AI Lab},
  198. year={2026},
  199. eprint={2605.06230},
  200. archivePrefix={arXiv},
  201. primaryClass={cs.AI},
  202. url={https://arxiv.org/abs/2605.06230},
  203. }
  204. ```

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.