SAfactory
SAfactory: A Scalable Agentic Infrastructure for Training Trustworthy Autonomous Intelligence
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 11 Nov 2026.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/AI45Lab/SAfactory with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/AI45Lab/SAfactory'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/ai45lab--safactory)HTML
<a href="/listings/ai45lab--safactory"><img src="/api/listings/ai45lab--safactory/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
1. SAfactory Installation And Gateway Configuration
Before running Geo3K, prepare the runtime image and dataset as described in [Standard Environment: Geo3K](docs/reference/environments.md#standard-environment-geo3k).
Install SAfactory:
git clone https://github.com/AI45Lab/SAfactory.git cd SAfactory pip install -U -r requirements.txt
Docker must be available locally, and the current user must be able to run `docker build`, `docker run`, and `docker exec`.
Create a local Gateway config:
cp gateway/config.example.yaml gateway/config.local.yaml
Edit `gateway/config.local.yaml` and explicitly set the storage path and model route:
listen_host: 0.0.0.0
listen_port: 8000
base_session_path: /v1/sessions
max_steps: -1
storage_type: sqlite
storage_config:
db_url: sqlite://env_trajs.db
llm_routes:
geo3k_model:
base_url: http://YOUR_LLM_HOST/v1
api_key: YOUR_API_KEY
supports_stream: true
max_concurrency: 64Start the Gateway:
python -m gateway --config gateway/config.local.yaml
Check readiness from another terminal:
curl http://127.0.0.1:8000/readyz
When using Cloud storage, the recommended configuration is to omit `landing_table` and `env_config_table` from the Gateway YAML and set only one profile in `.env`:
WT_SDK_PROFILE=test
or:
WT_SDK_PROFILE=production
The `test` profile selects `v2_landing_test` and `env_config_test`; the `production`/`prod` profile selects `wind_tunnel_landing` and `evaluation_env_config`. This keeps trajectory and environment-config writes in the same environment. Explicit table names in `storage_config` override the profile and should be used only for intentional special cases. SAfactory does not access the serving table.
2. Minimal Evaluation With Geo3K
Run a minimal Geo3K evaluation in Docker mode:
python launcher.py \ --mode docker \ --agent-config env/geo3k/geo3k_config.yaml \ --agent-start-config env/geo3k/geo3k_start.yaml \ --gateway-base-url http://127.0.0.1:8000/v1/sessions \ --llm-model geo3k_model \ --enable-evaluation \ --db-path sqlite://env_trajs.db \ --job-id geo3k-docker-smoke \ --pool-size 1 \ --max-workers 1 \ --max-steps 10
`--llm-model` must match a key under `llm_routes`. With `--enable-evaluation`, SAfactory calls `env/geo3k/rule_evaluator.py` and writes the final reward.
3. Minimal Training With Geo3K
Geo3K training uses the RL bridge under `rl/` and the example config at `rl/examples/geo3k_vl/env.sh`.
Before starting, edit `rl/examples/geo3k_vl/env.sh`, or export the same variables in each terminal. Set the required local paths and scale the first run down:
export AIEVOBOX_ROOT=$(pwd)
export AIEVOBOX_MODE=docker
export AIEVOBOX_AGENT_CONFIG=${AIEVOBOX_ROOT}/env/geo3k/geo3k_config.yaml
export AIEVOBOX_AGENT_START_CONFIG=${AIEVOBOX_ROOT}/env/geo3k/geo3k_start.yaml
export AIEVOBOX_GATEWAY_HOST=127.0.0.1
export AIEVOBOX_GATEWAY_PORT=8000
export RL_MODEL=geo3k_model
export AIEVOBOX_POOL_SIZE=2
export RL_GROUP_SIZE=2
export RL_EPOCH=1
export HF_CKPT_DIR=/path/to/hf-checkpoint
export SLIME_HOME=/path/to/slime
export MEGATRON_HOME=/path/to/Megatron-LMThen start two processes from the repository root:
RL_ENV_SH=rl/examples/geo3k_vl/env.sh bash rl/run_slime_generator.sh
RL_ENV_SH=rl/examples/geo3k_vl/env.sh bash rl/run_buffer_server.sh
Rewards for RL training come from completed trajectories in the database: `rule_evaluator.py` writes the reward after evaluation, and Buffer Server fetches trajectory rows with rewards from the same database used by Launcher / Gateway before serving them to the training process.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 278 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- <div align="center">
- # SAfactory
- <p align="center">
- <a href="README_CN.md">中文</a> | English
- </p>
- **SAfactory is a scalable infrastructure for agent evaluation, trajectory collection, and reinforcement learning training. It schedules agents and benchmark environments as external runtimes, routes model calls through a session-aware OpenAI-compatible Gateway, records trajectories, and feeds completed rollouts into training systems such as Slime.**
- <p align="center">
- <a href="#why-safactory">Why SAfactory</a> •
- <a href="#demo">Demo</a> •
- <a href="#agent-skill">Agent Skill</a> •
- <a href="#quick-start">Quick Start</a> •
- <a href="#documentation">Documentation</a> •
- <a href="#citation">Citation</a>
- </p>
- 
- 
- 
- 
- </div>
- ---
- ## <a id="why-safactory"></a>✨ Why SAfactory
- SAfactory provides one unified workflow for agent onboarding, benchmark onboarding, evaluation, rollout data generation, and RL training.
- The core runtime contract is:
- - one dataset row becomes one scheduled episode;
- - each episode gets its own `session_id` and Gateway session;
- - the runtime calls the target model through the Gateway;
- - the runtime returns one JSON result;
- - `rule_evaluator.py` converts runtime output and trajectory data into the SAfactory format;
- - completed trainable trajectories can be consumed by the RL Buffer Server or persisted as reusable data assets.
- ## <a id="demo"></a>🎬 Demo
- <div align="center">
- https://github.com/user-attachments/assets/4c551b27-ce4d-4fc8-8df6-d6dc8100cc88
- *Click to watch the full demo*
- </div>
- ## <a id="agent-skill"></a>🧩 Agent Skill Quick Start
- This repository includes an Agent skill for template-based environment integration, local contract tests, optional Docker/RJob evaluation, and GRPO/RL workflows:
- ```text
- skills/safactory-workflows/SKILL.md
- ```
- ### <a id="benchmark-onboarding-prompt"></a>Benchmark Onboarding Prompt
- Provide the benchmark source, one-row dataset schema, 1–2 representative cases, native single-case command, and native output format. Specify Docker or RJob deployment when known. **Integration alone does not require evaluation, a score/reward definition, a running Gateway, or an internal cluster.** Only provide scoring details when evaluation is requested.
- The Agent copies fixed runner/config templates, fills the environment hooks, and runs local contract tests with an owned mock model endpoint. Native dependencies must be available locally or represented by explicit test fixtures. Live deployment is a separate check when image/data/model/runtime access is available; a helper starts and stops Gateway without a second terminal. RJob live runs additionally require an existing cluster configuration and a cluster-reachable Gateway URL.
- <details>
- <summary>Expand to get the Benchmark Onboarding Prompt</summary>
- ```text
- Use skills/safactory-workflows to onboard this benchmark into SAfactory.
- [Goal]
- - evaluation: [disabled (default, integration only) / enabled]
- - target deployment mode: [docker / rjob / decide after adapter inspection]
- - validation: [local contract first / also run live deployment when available]
- [Benchmark]
- - environment name: ____________________
- - source or checkout path/repository: ____________________
- - dataset path and one-row schema: ____________________
- - 1–2 case IDs/rows: ____________________
- - native single-case command, or README file/section: ____________________
- - Docker image, if available: ____________________
- - native result/output path and format: ____________________
- [Only if evaluation is enabled]
- - native score field/file, range, meaning, and pass condition: ____________________
- Start from the fixed templates based on docs/guides/custom-environment.md.
- Keep protocol handling in runner.py and fill adapter.py with single-case row
- mapping, Gateway model routing, native command execution, and output collection.
- Do not reimplement native benchmark solving/scoring logic.
- Only add rule_evaluator.py and --enable-evaluation if evaluation is enabled.
- Run local contract checks without requiring Gateway or RJob cluster setup.
- Report fixtures used, native output mapping, and which checks actually passed;
- keep local contract, live deployment, and evaluation results distinct.
- Ask only for missing information needed by the next dependent step.
- ```
- </details>
- See [Custom Environments](docs/guides/custom-environment.md) and the skill's [integration reference](skills/safactory-workflows/references/environment-integration.md) for the templates and Docker/RJob deployment settings. Verify any environment with one command — `python skills/safactory-workflows/scripts/check_environment.py --env env/<name>` — which runs static consistency checks, a contract smoke test against an owned mock endpoint, and an optional live stage, labeling each failure with the side that owns it. If your Agent cannot discover local skills automatically, include `skills/safactory-workflows/` explicitly in the prompt.
- ## <a id="quick-start"></a>🚀 Quick Start
- ### 1. SAfactory Installation And Gateway Configuration
- Before running Geo3K, prepare the runtime image and dataset as described in [Standard Environment: Geo3K](docs/reference/environments.md#standard-environment-geo3k).
- Install SAfactory:
- ```bash
- git clone https://github.com/AI45Lab/SAfactory.git
- cd SAfactory
- pip install -U -r requirements.txt
- ```
- Docker must be available locally, and the current user must be able to run `docker build`, `docker run`, and `docker exec`.
- Create a local Gateway config:
- ```bash
- cp gateway/config.example.yaml gateway/config.local.yaml
- ```
- Edit `gateway/config.local.yaml` and explicitly set the storage path and model route:
- ```yaml
- listen_host: 0.0.0.0
- listen_port: 8000
- base_session_path: /v1/sessions
- max_steps: -1
- storage_type: sqlite
- storage_config:
- db_url: sqlite://env_trajs.db
- llm_routes:
- geo3k_model:
- base_url: http://YOUR_LLM_HOST/v1
- api_key: YOUR_API_KEY
- supports_stream: true
- max_concurrency: 64
- ```
- Start the Gateway:
- ```bash
- python -m gateway --config gateway/config.local.yaml
- ```
- Check readiness from another terminal:
- ```bash
- curl http://127.0.0.1:8000/readyz
- ```
- When using Cloud storage, the recommended configuration is to omit
- `landing_table` and `env_config_table` from the Gateway YAML and set only one
- profile in `.env`:
- ```bash
- WT_SDK_PROFILE=test
- ```
- or:
- ```bash
- WT_SDK_PROFILE=production
- ```
- The `test` profile selects `v2_landing_test` and `env_config_test`; the
- `production`/`prod` profile selects `wind_tunnel_landing` and
- `evaluation_env_config`. This keeps trajectory and environment-config writes
- in the same environment. Explicit table names in `storage_config` override the
- profile and should be used only for intentional special cases. SAfactory does
- not access the serving table.
- ### 2. Minimal Evaluation With Geo3K
- Run a minimal Geo3K evaluation in Docker mode:
- ```bash
- python launcher.py \
- --mode docker \
- --agent-config env/geo3k/geo3k_config.yaml \
- --agent-start-config env/geo3k/geo3k_start.yaml \
- --gateway-base-url http://127.0.0.1:8000/v1/sessions \
- --llm-model geo3k_model \
- --enable-evaluation \
- --db-path sqlite://env_trajs.db \
- --job-id geo3k-docker-smoke \
- --pool-size 1 \
- --max-workers 1 \
- --max-steps 10
- ```
- `--llm-model` must match a key under `llm_routes`. With `--enable-evaluation`, SAfactory calls `env/geo3k/rule_evaluator.py` and writes the final reward.
- ### 3. Minimal Training With Geo3K
- Geo3K training uses the RL bridge under `rl/` and the example config at `rl/examples/geo3k_vl/env.sh`.
- Before starting, edit `rl/examples/geo3k_vl/env.sh`, or export the same variables in each terminal. Set the required local paths and scale the first run down:
- ```bash
- export AIEVOBOX_ROOT=$(pwd)
- export AIEVOBOX_MODE=docker
- export AIEVOBOX_AGENT_CONFIG=${AIEVOBOX_ROOT}/env/geo3k/geo3k_config.yaml
- export AIEVOBOX_AGENT_START_CONFIG=${AIEVOBOX_ROOT}/env/geo3k/geo3k_start.yaml
- export AIEVOBOX_GATEWAY_HOST=127.0.0.1
- export AIEVOBOX_GATEWAY_PORT=8000
- export RL_MODEL=geo3k_model
- export AIEVOBOX_POOL_SIZE=2
- export RL_GROUP_SIZE=2
- export RL_EPOCH=1
- export HF_CKPT_DIR=/path/to/hf-checkpoint
- export SLIME_HOME=/path/to/slime
- export MEGATRON_HOME=/path/to/Megatron-LM
- ```
- Then start two processes from the repository root:
- ```bash
- RL_ENV_SH=rl/examples/geo3k_vl/env.sh bash rl/run_slime_generator.sh
- ```
- ```bash
- RL_ENV_SH=rl/examples/geo3k_vl/env.sh bash rl/run_buffer_server.sh
- ```
- Rewards for RL training come from completed trajectories in the database: `rule_evaluator.py` writes the reward after evaluation, and Buffer Server fetches trajectory rows with rewards from the same database used by Launcher / Gateway before serving them to the training process.
- Buffer Server can automatically start a Gateway and route `RL_MODEL` to the Slime-hosted LLM proxy. If a manually started Gateway is already using the same port, stop it first. Set `AIEVOBOX_GATEWAY_AUTOSTART=0` only when the external Gateway already has the correct route and storage configuration.
- ## <a id="documentation"></a>📚 Documentation
- ### Guides
- | Guide | What it covers |
- |-------|----------------|
- | [Custom Environments](docs/guides/custom-environment.md) | How to onboard a new external runtime adapter. |
- | [Evaluation](docs/guides/evaluation.md) | Rule evaluator discovery, interfaces, reward writing, and Geo3K evaluation. |
- | [RL Training](docs/guides/rl-training.md) | Buffer Server, Slime generator, Geo3K training path, and key variables. |
- | [Data Manager](docs/guides/data-manager.md) | SQLite storage behavior, table schema, row types, and query examples. |
- | [S3 + LanceDB Storage](docs/guides/S3+LanceDB-storage.md) | How to switch trajectory storage from local SQLite to S3 + LanceDB. |
- ### Internal
- | Internal Doc | What it covers |
- |--------------|----------------|
- | [RJob Mode](docs/internal/rjob-mode.md) | Remote RJob runtime config, authentication, mounts, Gateway reachability, and Geo3K examples. |
- | [Sandbox Mode](docs/internal/sandbox-mode.md) | Brainbox Sandbox Environment config, volumes, lifecycle, and launch flow. |
- ### Reference
- | Reference | What it covers |
- |-----------|----------------|
- | [CLI And Configuration](docs/reference/configuration.md) | Launcher flags, Gateway config, agent config, start config, RJob, and Sandbox settings. |
- | [Supported Environments](docs/reference/environments.md) | Checked-in adapters, the standard Geo3K path, and the runtime matrix. |
- | [Gateway Reference](docs/reference/gateway.md) | OpenAI-compatible routes, session endpoints, telemetry, request logs, and storage consistency. |
- | [Report](https://arxiv.org/pdf/2605.06230) | SAfactory report. |
- ## <a id="citation"></a>📖 Citation
- If SAfactory or SAfactory-generated datasets are useful in your work, please cite this repository and the specific dataset or report you used.
- ```bibtex
- @misc{chen2026safactoryscalableagenticinfrastructure,
- title={SAfactory: A Scalable Agentic Infrastructure for Training Trustworthy Autonomous Intelligence},
- author={Shanghai AI Lab},
- year={2026},
- eprint={2605.06230},
- archivePrefix={arXiv},
- primaryClass={cs.AI},
- url={https://arxiv.org/abs/2605.06230},
- }
- ```
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.