Skip to content
Back to search

RoboLab

A simulation benchmarking platform for evaluating generalist robot policies

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/NVlabs/RoboLab with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/NVlabs/RoboLab'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/nvlabs--robolab/badge.svg)](/listings/nvlabs--robolab)

HTML

<a href="/listings/nvlabs--robolab"><img src="/api/listings/nvlabs--robolab/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Requires [uv](https://docs.astral.sh/uv/getting-started/installation/) and a system `ffmpeg` (used for video recording). The IsaacSim/IsaacLab stack is selected at install time via a mutually-exclusive extra — `isaac50` (IsaacSim 5.0 / IsaacLab 2.2.0, default) or `isaac51` (IsaacSim 5.1 / IsaacLab 2.3.2.post1). See [Requirements](#requirements) for hardware.

Installation

sudo apt install ffmpeg
git clone <repo_url>
cd robolab
uv venv --python 3.11
source .venv/bin/activate
uv sync --extra isaac50          # IsaacSim 5.0 / IsaacLab 2.2.0 (default)
# uv sync --extra isaac51        # IsaacSim 5.1 / IsaacLab 2.3.2.post1

The two stacks cannot coexist in one environment. To keep both available, install each into its own venv via `UV_PROJECT_ENVIRONMENT`:

UV_PROJECT_ENVIRONMENT=.venv    uv sync --extra isaac50
UV_PROJECT_ENVIRONMENT=.venv-51 uv sync --extra isaac51

Verify installation:

uv run pytest tests/

This runs the install-verification suite end-to-end: isaaclab importable, all task definitions valid, env factory populated, one full episode runs. The suite auto-accepts the NVIDIA Omniverse EULA so the run is fully headless with no prompts. More details at [Debugging → Diagnostic Scripts](docs/debug.md#diagnostic-scripts).

> **Running without activating the venv**: if you don't `source .venv/bin/activate`, prefix every `python` command with `uv run` (e.g. `uv run pytest tests/`).

> **EULA outside the test suite**: when running other entry points (e.g. `policies/pi0_family/run.py`) for the first time, set `export OMNI_KIT_ACCEPT_EULA=Y` once. Cached after first acceptance.

Run without a policy

# Run an empty episode with random actions
python examples/run_empty.py --headless

# Playback recorded demonstration data
python examples/run_recorded.py --headless

# Toggle the gripper open/closed while holding the arm fixed (sanity-check
# the gripper action path; saves sensor + viewport video to
# output/run_gripper_toggle/<task>/)
python examples/run_gripper_toggle.py --task BananaInBowlTask --headless

> **Replay**: `run_recorded.py` restores the recorded initial state, replays the recorded actions open-loop, and by default replays with the exact env configuration saved next to the recording (`env_cfg.json`). Note that the recorded outcome is not invariant across simulator versions — contact dynamics evolve between IsaacSim/IsaacLab releases (see [Requirements](#requirements)) — and faithful reproduction requires recording and replaying with a single env. See **[Replaying Recorded Episodes](docs/replay.md)** for the full guide, including replaying your own recordings, `--env-config`, and `--validate-states`.

Run with a policy

RoboLab uses a **server-client architecture**: your model runs as a standalone server, and RoboLab connects to it via a lightweight inference client. To quickly test RoboLab, try [Pi0.5 via OpenPI](policies/pi0_family/README.md).

Quick run after install in the RoboLab terminal, to see it working:

cd robolab
uv run python policies/pi0_family/run.py --policy pi05 --task BananaInBowlTask --num-envs 10

Use the [dashboard](#dashboard) to view the output written to your local folder.

Common CLI Options

# Run headlessly
python policies/pi0_family/run.py --policy pi05 --headless

# Run on specific tasks (these two are good for sanity checking)
python policies/pi0_family/run.py --policy pi05 --task BananaInBowlTask RubiksCubeAndBananaTask

# Run on a tag of tasks
python policies/pi0_family/run.py --policy pi05 --tag semantics

# Run 12 parallel episodes per task
python policies/pi0_family/run.py --policy pi05 --headless --num-envs 12

# Disable subtask progress tracking (on by default; drops score/reason from results)
python policies/pi0_family/run.py --policy pi05 --disable-subtask

# Resume a previous run (skips completed episodes)
python policies/pi0_family/run.py --policy pi05 --output-folder-name my_previous_run

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 214 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <h1><picture><source media="(prefers-color-scheme: dark)" srcset="docs/images/ribble.gif"><img src="docs/images/ribble-dark.gif" alt="" height="42" align="absmiddle" /></picture> RoboLab</h1>
  2. [🌐 Website](https://research.nvidia.com/labs/srl/projects/robolab) · [📄 Paper](https://arxiv.org/abs/2604.09860) · [🏆 Leaderboard](https://research.nvidia.com/labs/srl/projects/robolab/leaderboard.html)
  3. **RoboLab** is a task-based evaluation benchmark for robot manipulation policies built on [NVIDIA Isaac Lab](https://github.com/isaac-sim/IsaacLab). It provides 100+ manipulation tasks with automated success detection, a server-client policy architecture, and multi-environment parallel evaluation — designed for reproducible, large-scale benchmarking of generalist robot policies in simulation.
  4. <h4 align="center"><a href="assets/objects/README.md">Objects</a> · <a href="assets/scenes/README.md">Scenes</a> · <a href="robolab/tasks/README.md">Tasks</a> · <a href="robolab/robots/README.md">Robots</a> · <a href="policies/README.md">Policy Clients</a></h4>
  5. <div align="center">
  6. <img src="docs/images/robolab.png" alt="RoboLab Overview" width="800"/>
  7. </div>
  8. ## News
  9. - **[2026/11/15]** 🚀 **v0.3.3** adds [initial-pose randomization](docs/init_pose_randomization.md) (`--init-pose-noise`) for a different collision-checked starting layout every episode, nine [lighting presets](docs/lighting.md), and GPU details in episode recordings. See the [Changelog](CHANGELOG.md).
  10. - **[2026/10/27]** 🚀 Added opt-in support for **Isaac Lab 3.0 beta and Isaac Sim 6.0.1**, including simulator compatibility updates and DROID rendering fixes. Isaac Sim 5.0 remains the default. See the [setup guide](docs/isaac_sim_6.md). Thanks to Meenal Parakh, Yannick Metz, and Finn Metz!
  11. - **[2026/10/18]** 🔥 Added [**Astranomy**](https://research.nvidia.com/labs/srl/projects/astranomy/) policy code, using general-purpose models (such as GPT-6 Astra) as a native robot policy.
  12. - **[2026/10/13]** 🔥 [**LADDER**](https://ladder.insait.ai/) introduces RoboLab-Long, a suite of 44 tasks for evaluating long-horizon manipulation, as part of a 145-task benchmark spanning three simulation suites. See [Task Libraries](docs/ecosystem.md#task-libraries).
  13. - **[2026/08/19]** 🤖 [**DROID+**](https://github.com/NVlabs/DroidPlus) is released: A simple DROID hardware stack. Your RoboLab policy setup will also work with DROID+.
  14. - **[2026/08/07]** 🔥 [**RoboVoLo**](https://github.com/NVlabs/RoboVoLo) is released — a task library from the [VoLo](https://chicychen.github.io/VoLo/) project extending RoboLab with long-horizon and reasoning-heavy manipulation tasks, evaluated via the bundled [VoLo policy backend](policies/volo/README.md).
  15. For all updates, including minor improvements and bug fixes, see the [Changelog](CHANGELOG.md).
  16. ## Key Features
  17. - **RoboLab-120**: An initial set of 120 brand new benchmark [tasks](robolab/tasks/README.md) spanning pick-and-place, stacking, rearrangement, tool use, and more — each with language instructions and automated success/failure detection via composable predicates.
  18. - **Bring your own robot**: Tasks are not tied to a specific robot embodiment, so you can plug in any robot compatible with IsaacLab!
  19. - **Rich Asset Libraries**: See a list of [objects](assets/objects/README.md), [scenes](assets/scenes/README.md), and curated [backgrounds](assets/backgrounds/README.md) — everything you need to create new scenes and new tasks for your own evaluation needs.
  20. - **AI-Enabled Workflows**: Generate new scenes and tasks **in minutes** using natural language with the [/robolab-scenegen](skills/robolab-scenegen/) and [/robolab-taskgen](skills/robolab-taskgen/) Claude Code skills.
  21. - **Selectable Lighting**: Nine [lighting presets and visual choice cards](docs/lighting.md#lighting-choice-cards), configurable through `lighting_selection=` in the Python registration API. Policy runners keep DROID's original lighting by default.
  22. - **Multi-Environment Parallel Evaluation**: Run multiple episodes in parallel across environments with vectorized conditionals and per-environment termination.
  23. - **Results Dashboard with Episode Videos and Cross-Experiment Analysis**: A self-contained web [dashboard](docs/dashboard.md) for browsing scenes/tasks, replaying episode videos, and comparing results across experiments.
  24. See the [Ecosystem](docs/ecosystem.md) page for projects built on RoboLab.
  25. ## Getting Started
  26. Requires [uv](https://docs.astral.sh/uv/getting-started/installation/) and a system `ffmpeg` (used for video recording). The Isaac Sim/Isaac Lab stack is selected with a mutually-exclusive extra: `isaac50` (the default), `isaac51`, or `isaac60`. See [Isaac Sim 6 support](docs/isaac_sim_6.md) and [Requirements](#requirements).
  27. ### Installation
  28. ```bash
  29. sudo apt install ffmpeg
  30. git clone <repo_url>
  31. cd robolab
  32. uv venv --python 3.11
  33. source .venv/bin/activate
  34. uv sync --extra isaac50 # Isaac Sim 5.0 / Isaac Lab 2.2.0 (default)
  35. # uv sync --extra isaac51 # Isaac Sim 5.1 / Isaac Lab 2.3.2.post1 (Python 3.11)
  36. # uv sync --python 3.12 --extra isaac60 # Isaac Sim 6.0.1 / Isaac Lab 3.0 beta
  37. ```
  38. The stacks cannot coexist in one environment. To keep multiple versions available, install each into its own venv:
  39. ```bash
  40. UV_PROJECT_ENVIRONMENT=.venv-50 uv sync --python 3.11 --extra isaac50
  41. UV_PROJECT_ENVIRONMENT=.venv-51 uv sync --python 3.11 --extra isaac51
  42. UV_PROJECT_ENVIRONMENT=.venv-60 uv sync --python 3.12 --extra isaac60
  43. ```
  44. Verify installation:
  45. ```bash
  46. uv run pytest tests/
  47. ```
  48. This runs the install-verification suite end-to-end: isaaclab importable, all task definitions valid, env factory populated, one full episode runs. The suite auto-accepts the NVIDIA Omniverse EULA so the run is fully headless with no prompts. More details at [Debugging → Diagnostic Scripts](docs/debug.md#diagnostic-scripts).
  49. > **Running without activating the venv**: if you don't `source .venv/bin/activate`, prefix every `python` command with `uv run` (e.g. `uv run pytest tests/`).
  50. > **EULA outside the test suite**: when running other entry points (e.g. `policies/pi0_family/run.py`) for the first time, set `export OMNI_KIT_ACCEPT_EULA=Y` once. Cached after first acceptance.
  51. ### Run without a policy
  52. ```bash
  53. # Run an empty episode with random actions
  54. python examples/run_empty.py --headless
  55. # Playback recorded demonstration data
  56. python examples/run_recorded.py --headless
  57. # Toggle the gripper open/closed while holding the arm fixed (sanity-check
  58. # the gripper action path; saves sensor + viewport video to
  59. # output/run_gripper_toggle/<task>/)
  60. python examples/run_gripper_toggle.py --task BananaInBowlTask --headless
  61. ```
  62. > **Replay**: `run_recorded.py` restores the recorded initial state, replays the recorded actions open-loop, and by default replays with the exact env configuration saved next to the recording (`env_cfg.json`). Note that the recorded outcome is not invariant across simulator versions — contact dynamics evolve between IsaacSim/IsaacLab releases (see [Requirements](#requirements)) — and faithful reproduction requires recording and replaying with a single env. See **[Replaying Recorded Episodes](docs/replay.md)** for the full guide, including replaying your own recordings, `--env-config`, and `--validate-states`.
  63. ### Run with a policy
  64. RoboLab uses a **server-client architecture**: your model runs as a standalone server, and RoboLab connects to it via a lightweight inference client. To quickly test RoboLab, try [Pi0.5 via OpenPI](policies/pi0_family/README.md).
  65. Quick run after install in the RoboLab terminal, to see it working:
  66. ```bash
  67. cd robolab
  68. uv run python policies/pi0_family/run.py --policy pi05 --task BananaInBowlTask --num-envs 10
  69. ```
  70. Use the [dashboard](#dashboard) to view the output written to your local folder.
  71. ### Common CLI Options
  72. ```bash
  73. # Run headlessly
  74. python policies/pi0_family/run.py --policy pi05 --headless
  75. # Run on specific tasks (these two are good for sanity checking)
  76. python policies/pi0_family/run.py --policy pi05 --task BananaInBowlTask RubiksCubeAndBananaTask
  77. # Run on a tag of tasks
  78. python policies/pi0_family/run.py --policy pi05 --tag semantics
  79. # Run 12 parallel episodes per task
  80. python policies/pi0_family/run.py --policy pi05 --headless --num-envs 12
  81. # Disable subtask progress tracking (on by default; drops score/reason from results)
  82. python policies/pi0_family/run.py --policy pi05 --disable-subtask
  83. # Resume a previous run (skips completed episodes)
  84. python policies/pi0_family/run.py --policy pi05 --output-folder-name my_previous_run
  85. # Randomize initial object poses: move each object up to 10 cm and turn it up to 60 degrees,
  86. # a different layout every episode (see docs/init_pose_randomization.md)
  87. python policies/pi0_family/run.py --policy pi05 --init-pose-noise 0.1
  88. ```
  89. ## Example Tasks
  90. See the full [Benchmark Task Library](robolab/tasks/README.md) for all 120 tasks.
  91. <div align="center">
  92. <img src="docs/images/Make_sure_all_the_white_mugs_are_upright_so_that_the_opening_is_facing_upwards_0_hstack_3X_fps24_width800.gif" alt="Make sure all the white mugs are upright so that the opening is facing upwards" width="800"/>
  93. <br><em>"Make sure all the white mugs are upright so that the opening is facing upwards."</em>
  94. </div>
  95. <div align="center">
  96. <img src="docs/images/Put_all_plastic_bottles_away_in_the_bin_3_hstack_3X_fps24.gif" alt="Put all plastic bottles away in the bin" width="800"/>
  97. <br><em>"Put all plastic bottles away in the bin."</em>
  98. </div>
  99. <div align="center">
  100. <img src="docs/images/Put_the_orange_measuring_cup_and_the_blue_measuring_cup_outside_of_the_plate_0_hstack_3X_fps24_width800.gif" alt="Put the orange measuring cup and the blue measuring cup outside of the plate" width="800"/>
  101. <br><em>"Put the orange measuring cup and the blue measuring cup outside of the plate."</em>
  102. </div>
  103. ## Dashboard
  104. A self-contained web dashboard for browsing the benchmark (scenes and tasks) and analyzing your experiment results.
  105. ```bash
  106. uv run robolab-dashboard
  107. # open http://localhost:8080
  108. ```
  109. <video src="https://github.com/user-attachments/assets/5992e61b-9043-4602-8402-04459da38421" autoplay controls muted loop playsinline width="800">
  110. Your viewer doesn't render inline video — see <a href="https://github.com/user-attachments/assets/5992e61b-9043-4602-8402-04459da38421">robolab_dashboard.mp4</a>.
  111. </video>
  112. See **[docs/dashboard.md](docs/dashboard.md)** for the full feature tour, CLI
  113. flags, and the API endpoints under the hood.
  114. ## Documentation
  115. Full documentation is at **[docs/README.md](docs/README.md)**, covering:
  116. - [Objects](docs/objects.md), [Scenes](docs/scene.md), [Tasks](docs/task.md) — Creating and managing assets and benchmark tasks
  117. - [Robots](docs/robots.md), [Cameras](docs/camera.md), [Lighting](docs/lighting.md), [Backgrounds](docs/background.md) — Configuring simulation parameters
  118. - [Environment Registration](docs/environment_registration.md) — Combining tasks with robot/observation/action configs
  119. - [Inference Clients](policies/README.md) — A list of supported open-source models and clients
  120. - [Replaying Recorded Episodes](docs/replay.md) — Playing back recorded HDF5 episodes faithfully
  121. - [Initial-Pose Randomization](docs/init_pose_randomization.md) — Giving each episode a different starting layout
  122. - [Analysis and Results](docs/analysis.md) — Summarizing, comparing, and auditing results
  123. - [Dashboard](docs/dashboard.md) — Interactive web viewer for benchmark, tasks, scenes, and eval results
  124. - [Subtask Checking](docs/subtask.md), [Conditionals](docs/task_conditionals.md), [Event Tracking](docs/event_tracking.md)
  125. - [Ecosystem](docs/ecosystem.md) — Task libraries and projects built on RoboLab
  126. ## Requirements
  127. | Dependency | Version |
  128. |---|---|
  129. | Isaac Sim | 5.0 (default), 5.1, or 6.0.1 |
  130. | Isaac Lab | 2.2.0 (default), 2.3.2.post1, or 3.0 beta 2 patch 1 |
  131. | Python | 3.12 for Isaac Sim 6; 3.11 for Isaac Sim 5 |
  132. | Linux | Ubuntu 22.04+ |
  133. > **Note on simulator versions**: the stacks ship different PhysX builds, so contact-rich dynamics (grasping, object settling) are not invariant across versions. Benchmark results are best compared on the same stack. RoboLab converts recorded quaternion layouts across versions, but numerical replay fidelity still requires the original simulator stack.
  134. - **Disk space**: ~8 GB (assets account for ~7 GB)
  135. - **GPU**: NVIDIA RTX GPU required. Recommend 48GB+ VRAM. See [Isaac Lab's hardware requirements](https://isaac-sim.github.io/IsaacLab/main/source/setup/installation/index.html#system-requirements) for recommended GPUs and VRAM.
  136. - **Speed**: 30 GPU hours / 100 tasks, 1.4 it/s (assuming ~200ms inference step)
  137. ## License
  138. The RoboLab framework is released under the [Apache License 2.0](./LICENSE).
  139. Third-party dependency licenses are listed in [THIRD_PARTY_NOTICES.md](./THIRD_PARTY_NOTICES.md).
  140. ## Citation
  141. ```bibtex
  142. @inproceedings{yang2026robolab,
  143. author = {Jenai Xuning Yang and Rishit Dagli and Alex Zook and Hugo Hadfield and Ankit Goyal and Stan Birchfield and Fabio Ramos and Jonathan Tremblay},
  144. title = {{RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies}},
  145. booktitle = {Proceedings of Robotics: Science and Systems},
  146. year = {2026},
  147. address = {Sydney, Australia},
  148. month = {July},
  149. url = {https://arxiv.org/abs/2604.09860}
  150. }
  151. ```
  152. ## Contributing
  153. See [CONTRIBUTING.md](CONTRIBUTING.md) for acknowledgements, issues, and how to contribute.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.