Skip to content
Back to search

graspgenx

Run and operate GraspGenX — a cross-embodiment foundation model for 6-DOF grasp generation. Use when installing GraspGenX (uv or Docker), running inference demos (segmented object point cloud, scene point cloud, or object mesh), choosing a gripper or planner (graspmoe vs diffusion), starting or calling the ZMQ client-server, driving grasping through the MCP server, integrating a new gripper (config wizard / sweep-volume params), or wiring grasps into an end-to-end cuRobo + Newton pick-and-place pipeline. Also covers checkpoint/asset setup and the depth-quality requirement for real-world use.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/nvlabs--graspgenx/badge.svg)](/listings/nvlabs--graspgenx)

HTML

<a href="/listings/nvlabs--graspgenx"><img src="/api/listings/nvlabs--graspgenx/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Choose your preferred installation method. For training, we recommend **Docker**. For inference, **uv** is sufficient.

| Method | Use Case | Complexity | |--------|----------|------------| | **Docker** | Training + Inference | ⭐⭐⭐ | | **uv** | Inference | ⭐ |

Installation with uv

Install [uv](https://docs.astral.sh/uv/getting-started/installation/). Eager PyTorch inference is the base install:

git clone https://github.com/NVlabs/GraspGenX.git && cd GraspGenX
uv sync

Fast inference is a separate native C++ runtime. Build it on the target GPU:

uv run --quiet --extra fast-inference graspgenx-install-fast-inference \
    --diffusion-grasps 100 \
    --obb-grasps 50

The default encoder profile is `[3500,2048,1024,512,512]`. Use `--ptv3-capacities INPUT POOL1 POOL2 POOL3 POOL4` to build another profile. See [capacity profiles](cpp_runtime/README.md#configurable-ptv3-capacities) for supported limits and custom artifact directories.

To install fast inference together with the Viser and rendering dependencies:

uv sync --quiet --extra fast-inference --extra visualization
uv run --quiet --extra fast-inference --extra visualization graspgenx-install-fast-inference \
    --diffusion-grasps 100 \
    --obb-grasps 50

The installer command installs the Python build dependencies, downloads the released checkpoints when needed, exports GPU-specific TensorRT plans, builds the C++ runtime into the package search path, and runs an end-to-end smoke test. It reuses complete plans and incremental CMake output on subsequent runs. The installer shows only phase summaries by default and prints captured tool output if a command fails. Add `--verbose` to stream ONNX, TensorRT, and CMake output.

TensorRT plan generation uses builder optimization level 2 by default for a reasonable install time. Add `--tensorrt-optimization-level 3` or `--tensorrt-optimization-level 5` to spend more build time searching for faster plans. Add `--force-artifacts` as well when rebuilding existing plans.

The host still needs a CUDA toolkit with `nvcc` and matching TensorRT runtime and development headers (`libnvinfer` and `NvInfer.h`). For a nonstandard CUDA or TensorRT installation, pass `--cuda-root` or `--tensorrt-root`. On Jetson, the installer uses JetPack's system TensorRT Python package so plan generation matches the native TensorRT runtime exactly.

Inside an existing conda or Python environment, use `pip install -e .` for PyTorch inference. Use the uv commands above for the complete native runtime; they select the CUDA 13 PyTorch and TensorRT packages together.

Add `--extra visualization` only when running the Viser or rendering demos, `--extra usd` only for USD loading/export on supported platforms, and `--extra training` only for dataset and training utilities.

See [Fast inference](FAST_INFERENCE.md) for the optimized pipeline and the [native C++ TensorRT runtime](cpp_runtime/README.md) for build and API details.

Fast-Inference Performance

Measurements use the native C++ runtime, the released 20-step DDPM schedule, 3,500 model input points, 100 diffusion candidates, and 50 analytic OBB candidates. Each result reports the mean of three independent process means; each process performs 300 CUDA Graph warmup replays followed by 300 timed replays. The timed boundary starts with a CUDA-resident point cloud and ends with ranked, CUDA-resident grasps and scores.

| Target | 150 candidates (100 diffusion + 50 OBB) | |---|---:| | NVIDIA RTX 6000 Ada | 3.178 ms |

Every benchmark output contains the exact requested count, finite poses and scores, descending confidence, deterministic replay, and no PTV3 topology-capacity overflow.

These measurements use TensorRT 10.16.1.11, the default capacity profile, and a synthetic object. They exclude initialization, plan construction, and input transfers. See the [benchmark guide](cpp_runtime/README.md#validation-and-benchmark)

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 507 lines

View this file on GitHub

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. ---
  2. name: graspgenx
  3. description: >-
  4. Run and operate GraspGenX — a cross-embodiment foundation model for 6-DOF grasp
  5. generation. Use when installing GraspGenX (uv or Docker), running inference
  6. demos (segmented object point cloud, scene point cloud, or object mesh),
  7. choosing a gripper or planner (graspmoe vs diffusion), starting or calling the
  8. ZMQ client-server, driving grasping through the MCP server, integrating a new
  9. gripper (config wizard / sweep-volume params), or wiring grasps into an
  10. end-to-end cuRobo + Newton pick-and-place pipeline. Also covers checkpoint/asset
  11. setup and the depth-quality requirement for real-world use.
  12. ---
  13. # GraspGenX
  14. GraspGenX is a **cross-embodiment** grasp model: one model, conditioned on a
  15. gripper's *swept volume*, predicts 6-DOF grasps for **any** gripper — including
  16. grippers unseen in training. Output is a set of `(K, 4, 4)` SE(3) grasp poses
  17. plus `(K,)` confidence scores; higher score = better, and visualizers color
  18. grasps red→green by score.
  19. **Repo layout (what lives where):**
  20. | Path | Purpose |
  21. |------|---------|
  22. | `graspgenx/` | The package: model, samplers, `serving/` (ZMQ), `x_grippers.py`, utils |
  23. | `scripts/` | Demos, `list_grippers.py`, `gripper_config_wizard.py`, client-server examples |
  24. | `client-server/` | ZMQ inference server + CLI client |
  25. | `mcp/` | MCP server bridging LLM tool-calls → the ZMQ server |
  26. | `end2end/` | GraspGenX → cuRobo → Newton/MuJoCo pick-and-place pipeline |
  27. | `assets/` | `sample_data/`, `proc_grippers/` |
  28. | `ext/` | Auto-cloned checkpoints + gripper descriptions (see below) |
  29. **Convention:** run every command with `uv run python …` from the repo root.
  30. ## Checkpoints & gripper assets (automatic)
  31. On the **first `import graspgenx`**, two Hugging Face repos are shallow-cloned
  32. (needs `git` + `git-lfs` on PATH) into `ext/`:
  33. - Model checkpoints → `ext/graspgenx_checkpoints/release/{gen,dis}/`
  34. (each has `config.yaml` + `epoch_*.pth`).
  35. - Gripper descriptions → `ext/gripper_descriptions/`.
  36. Override the locations if you already have them elsewhere:
  37. ```bash
  38. export GRASPGENX_CHECKPOINT_DIR=/path/to/graspgenx_checkpoints
  39. export GRASPGENX_GRIPPER_CFG_DIR=/path/to/gripper_descriptions # must already exist
  40. ```
  41. Verify the assets resolve:
  42. ```bash
  43. uv run python -c "from graspgenx import get_gripper_descriptions_root; print(get_gripper_descriptions_root())"
  44. uv run python scripts/list_grippers.py # all available grippers
  45. ```
  46. ---
  47. ## 1. Installation
  48. Pick **uv** for inference, **Docker** for training (Docker also does inference).
  49. ### uv (recommended for inference)
  50. ```bash
  51. git clone https://github.com/NVlabs/GraspGenX.git && cd GraspGenX && uv sync
  52. ```
  53. Python >= 3.10. `pip install -e .` also works inside a conda or venv. Add the
  54. `fast-inference` extra only for the native C++ runtime.
  55. **Optional extras** (add only what you need):
  56. | Command | Adds |
  57. |---------|------|
  58. | `uv sync --extra serve` | ZMQ client-server (`pyzmq`, `msgpack`, `msgpack-numpy`) |
  59. | `uv sync --extra fast-inference` | Build dependencies for the native C++ runtime |
  60. | `uv sync --extra training` | Training and dataset utilities |
  61. | `uv sync --extra visualization` | Viser demos and offline rendering |
  62. | `uv sync --extra end2end` | cuRobo + Newton/MuJoCo + CoACD + USD + fcl (pick-and-place) |
  63. | `uv sync --extra dev` | `black`, `isort`, `flake8` |
  64. ### Native inference
  65. Build the shared library and plans on the target GPU:
  66. ```bash
  67. uv run --quiet --extra fast-inference graspgenx-install-fast-inference \
  68. --diffusion-grasps 100 --obb-grasps 50
  69. ```
  70. The optimized runtime is implemented in `cpp_runtime/`; Python calls its C ABI.
  71. It uses DDPM20, 100 diffusion + 50 OBB candidates, and capacities
  72. `[3500,2048,1024,512,512]` by default. Custom counts and
  73. `--ptv3-capacities INPUT POOL1 POOL2 POOL3 POOL4` require matching plans.
  74. See [Fast inference](FAST_INFERENCE.md) for the supported input and execution
  75. contract. Add `--fast-inference --num_grasps 100` to the point-cloud demos to
  76. use the plans built above.
  77. ### Docker (training + inference)
  78. ```bash
  79. bash docker/build.sh # builds image x_grasp:3.0 and x_grasp:latest
  80. ```
  81. Base image `nvcr.io/nvidia/pytorch:25.03-py3`, `WORKDIR /code/`. The image has
  82. **no CMD/ENTRYPOINT and there is no docker-compose file** — mount the repo and
  83. run commands yourself:
  84. ```bash
  85. docker run --gpus all -v "$PWD":/code -it x_grasp:latest bash
  86. # inside: python scripts/demo_object_pc.py ... (uv is not required in Docker)
  87. ```
  88. For headless rendering (no display) in Docker or over SSH, prefix commands with
  89. `PYOPENGL_PLATFORM=egl`.
  90. ---
  91. ## 2. Inference inputs and controls
  92. Every inference picks three things:
  93. 1. **Input modality** — segmented object point cloud, full scene point cloud, or
  94. object mesh.
  95. 2. **`--gripper_name`** — one of 26 built-in grippers (see `list_grippers.py`):
  96. `robotiq_2f_85`, `robotiq_2f_140`, `robotiq_3f`, `robotiq_hande`,
  97. `franka_panda`, `franka_umi`, `inspire_hand`, `surge_hand`, `barrett_hand`,
  98. `unitree_g1`, `galaxea_g1`, `ezgripper`, `arx_x5`, `abb_yumi`, `bd_spot`,
  99. `dh_ag95`, `fetch_robot`, `onrobot_RG2`, `onrobot_RG6`, `piper_hand`,
  100. `sawyer_hand`, `schunk_wsg50`, `sharpa_wave`, `tesollo_delto2f`, `wuji_hand`,
  101. `xarm_hand`. (Plus any gripper you add — see §6.)
  102. 3. **`--planner`** — `graspmoe` (default; diffusion ∪ oriented-bounding-box
  103. grasps, all scored by the discriminator) or `diffusion` (diffusion sampler
  104. only, faster).
  105. **Visualization:** demos serve an interactive [viser](https://viser.studio/) GUI
  106. at **`http://localhost:8080`** (open it in a browser; click *Next* to cycle
  107. objects, top grasp shown in blue). The port is **fixed at 8080** — there is *no*
  108. `--port` flag on the demo scripts; if 8080 is taken, the actual port is printed
  109. in the log. Over SSH, forward it: `ssh -N -L 8080:localhost:8080 <host>`.
  110. Sample inputs under `assets/sample_data/`: `real_world/{00,01}/` (depth + seg +
  111. meta scenes), `object_pc/*.json`, `object_mesh/{banana,box}.obj`,
  112. `hope_objects/*.obj`.
  113. ---
  114. ## 3. Inference demos
  115. ### `demo_object_pc.py` — segmented object point clouds
  116. Runs inference on one pre-segmented object at a time.
  117. ```bash
  118. uv run python scripts/demo_object_pc.py \
  119. --sample_data_dir assets/sample_data/real_world \
  120. --gripper_name robotiq_2f_85 \
  121. --plot_top_mesh
  122. ```
  123. `--gripper_name` accepts **multiple** grippers (compare side by side). Common
  124. flags: `--planner {graspmoe,diffusion}`, `--grasp_threshold 0.7` (`-1.0` = keep
  125. top-k instead), `--num_grasps 200`, `--return_topk --topk_num_grasps 100`,
  126. `--plot_top_mesh` (gripper mesh at the top grasp), `--moe_obb_density
  127. {sparse,dense,dense-topandside}` (default `dense-topandside`).
  128. ### `demo_scene_pc.py` — objects within a full scene
  129. Loads a full-scene point cloud with per-object segmentation, runs inference on
  130. every object, and **filters grasps that collide with the rest of the scene**.
  131. ```bash
  132. uv run python scripts/demo_scene_pc.py \
  133. --sample_data_dir assets/sample_data/real_world \
  134. --gripper_name robotiq_2f_85 \
  135. --scene 00
  136. ```
  137. Collision filtering is **on by default**; disable with `--no-filter_collisions`.
  138. `--scene 00` restricts to a single `real_world` scene. `--plot_top_mesh` is on by
  139. default here (`--no-plot_top_mesh` to disable). Add `--fast-inference` to use
  140. the shared native runtime, with `--num_grasps` matching the installed plans.
  141. ### `demo_object_mesh.py` — object meshes
  142. Samples surface points from a mesh, runs inference, visualizes mesh + grasps.
  143. ```bash
  144. uv run python scripts/demo_object_mesh.py \
  145. --mesh_file assets/sample_data/object_mesh/banana.obj \
  146. --mesh_scale 1.0 \
  147. --gripper_name inspire_hand \
  148. --grasp_threshold -1.0 --return_topk --topk_num_grasps 100 --plot_top_mesh
  149. ```
  150. Formats: `.obj`, `.stl`, `.ply`, USD (`.usd/.usda/.usdc/.usdz`). For
  151. **headless/batch** (no viewer) that writes grasps to a YAML in the Isaac grasp
  152. format:
  153. ```bash
  154. uv run python scripts/demo_object_mesh.py --mesh_file <mesh> --gripper_name <g> \
  155. --no-visualization --output_file /tmp/grasps.yml
  156. ```
  157. ### Other inference entrypoints
  158. - `scripts/batch_inference_scene.py` — headless, one batched pass over a scene,
  159. optional `.npz` output (`--output_file`).
  160. - `scripts/inference_xgrasp.py` — Hydra-driven large-scale eval to H5 (config
  161. overrides, not argparse).
  162. ---
  163. ## 4. MCP server (LLM-driven grasping)
  164. `mcp/` is a lightweight bridge so an LLM (Claude Desktop, Cursor, VS Code) can
  165. call GraspGenX as tools. It needs **no CUDA and no model weights** — it forwards
  166. requests over ZMQ to a running GraspGenX server:
  167. ```
  168. LLM ──MCP (stdio)──▶ mcp/ bridge ──ZMQ (tcp)──▶ GraspGenX server (GPU)
  169. ```
  170. **Prerequisite:** the ZMQ server (§5) must already be running.
  171. Install and register:
  172. ```bash
  173. cd mcp && uv venv --python 3.10 .venv && source .venv/bin/activate && uv pip install -e .
  174. ```
  175. Tools exposed: `generate_grasps_from_mesh`, `generate_grasps_from_point_cloud`,
  176. `generate_grasps_from_sweep_volume` (params-only, no server assets),
  177. `visualize_grasps`, `graspgenx_health_check`, `graspgenx_server_info`. Being
  178. cross-embodiment, the grasp tools take a `gripper_name` per call.
  179. Client config (Cursor `.cursor/mcp.json` shown; Claude Desktop / VS Code are the
  180. same shape — see `mcp/README.md` for all three):
  181. ```json
  182. {
  183. "mcpServers": {
  184. "graspgenx": {
  185. "command": "uv",
  186. "args": ["run", "--directory", "/absolute/path/to/GraspGenX/mcp", "mcp-server-graspgenx"],
  187. "env": { "GRASPGENX_HOST": "localhost", "GRASPGENX_PORT": "5556" }
  188. }
  189. }
  190. }
  191. ```
  192. ---
  193. ## 5. ZMQ client-server
  194. Serve GraspGenX once (model loaded on GPU) and let any process — same machine or
  195. across the network, no GPU/torch needed client-side — request grasps.
  196. ### Start the server
  197. ```bash
  198. uv sync --extra serve # one-time: ZMQ deps
  199. uv run python client-server/graspgenx_server.py \
  200. --config ext/graspgenx_checkpoints/release \
  201. --assets_dir assets \
  202. --port 5556 \
  203. --default_gripper franka_panda # optional; used when a client omits gripper_name
  204. ```
  205. `--config` is the checkpoint **root** directory (the one containing `gen/` and
  206. `dis/`), *not* the inner `config.yaml`. Add `--fast-inference` to delegate the
  207. complete optimized pipeline to the native C++ runtime. The
  208. model loads eagerly at startup; per-gripper samplers load lazily and are cached.
  209. ### Wire actions (four modes)
  210. | Action | Input | Gripper spec | Threshold/top-k |
  211. |--------|-------|--------------|-----------------|
  212. | `infer` | object point cloud | `gripper_name` (name-based) | server-side |
  213. | `infer_object` | segmented object PC | sweep-volume params (12 numbers) | client-side |
  214. | `infer_scene_depth` | depth (m) + intrinsics + instance mask | sweep-volume params | client-side |
  215. | `infer_scene_pc` | scene PC + instance mask | sweep-volume params | client-side |
  216. The three sweep-volume modes return **all** generated grasps with scores and
  217. reject `gripper_name`/`grasp_threshold`/`topk` on the wire — the client applies
  218. thresholding/top-k. `health` → `{"status":"ok"}`; `metadata` → default/loaded
  219. grippers + model config.
  220. ### CLI client
  221. ```bash
  222. # General CLI client (mesh or point cloud → grasps, optional viser):
  223. uv run python client-server/graspgenx_client.py \
  224. --mesh_file assets/sample_data/object_mesh/box.obj --gripper_name franka_panda \
  225. --host localhost --port 5556 --visualize
  226. ```
  227. ### Python API
  228. ```python
  229. from graspgenx.serving import GraspGenXClient, SweepVolumeParams
  230. with GraspGenXClient(host="localhost", port=5556) as client:
  231. print(client.server_metadata)
  232. grasps, conf = client.infer(point_cloud, gripper_name="franka_panda") # (K,4,4), (K,)
  233. # params-only (no server assets for the gripper):
  234. params = SweepVolumeParams.from_gripper_config("franka_panda") # or build 12 numbers directly
  235. grasps, conf = client.infer_object(object_pc, params, planner="graspmoe")
  236. per_instance = client.infer_scene_pc(scene_pc, instance_mask, params) # {id: (grasps, conf)}
  237. ```
  238. `infer_scene_depth` / `infer_scene_pc` return a `{instance_id: (grasps, conf)}`
  239. dict; pass `return_branch_tags=True` to also get per-grasp `"diff"`/`"obb"` tags.
  240. ---
  241. ## 6. Integrating a new gripper
  242. GraspGenX runs zero-shot on any gripper given a URDF + meshes. The only thing it
  243. needs beyond a stock URDF is a `config.json` describing the open/close joint
  244. states and the **swept-volume boxes** that condition the model. An interactive
  245. wizard generates it.
  246. ### Run the wizard
  247. ```bash
  248. uv run python scripts/gripper_config_wizard.py \
  249. --urdf /path/to/your/gripper.urdf \
  250. --name <gripper_name> \
  251. --port 8081
  252. ```
  253. Open `http://localhost:8081` and step through six panels (click *Confirm* to
  254. advance): **1** align base frame (`+Z` = approach along the fingers, `+X` =
  255. closing direction) → **2** confirm fully-open joint pose → **3** drag the blue
  256. box to enclose the inner finger volume at open → **4** set the closed pose, drag
  257. the orange box for the half-open volume → **5** review the open↔close animation
  258. → **6** pick gripper type + symmetry and save.
  259. ### What gets written
  260. Into `<gripper_descriptions_root>/…/assets/x_grippers/<name>/` (root resolved via
  261. `GRASPGENX_GRIPPER_CFG_DIR`, else `ext/gripper_descriptions/`):
  262. - `gripper.urdf` + `meshes/` — copied from your source.
  263. - `vis_mesh.obj` — merged visual mesh at the open pose.
  264. - `config.json` — the model conditioning. Key fields:
  265. ```jsonc
  266. {
  267. "open": { "<joint>": <float>, ... }, // fully-open joint state
  268. "close": { "<joint>": <float>, ... }, // closed joint state
  269. "fingertip": [x, y, z], // fingertip point (z = depth)
  270. "sweep_volume": {
  271. "extents": [ex, ey, ez], "offset": [ox, oy, oz], // open box
  272. "extents2": [ex, ey, ez], "offset2": [ox, oy, oz] // half-open box
  273. },
  274. "type": "parallel_2f", // parallel_2f=0, revolute_2f=1, revolute_3f=2
  275. "base_rotation": [[4x4]], "symmetric": true, "bbox": [...], "standoff": [...]
  276. }
  277. ```
  278. Collision mesh, point-cloud, and TSDF caches are generated lazily on first use —
  279. nothing else to run.
  280. ### Verify, then use it
  281. ```bash
  282. uv run python scripts/vis_gripper_desc.py --gripper <gripper_name> --port 8081 # animates open↔close + boxes
  283. uv run python scripts/demo_object_pc.py \
  284. --sample_data_dir assets/sample_data/real_world \
  285. --gripper_name <gripper_name> --plot_top_mesh
  286. ```
  287. **Params-only alternative:** you can skip on-disk assets entirely and feed the
  288. gripper as 12 raw sweep-volume numbers at inference time via `SweepVolumeParams`
  289. (the `infer_object`/scene actions in §5, or the MCP
  290. `generate_grasps_from_sweep_volume` tool).
  291. If you onboard a gripper not already in the shared set, please upstream the saved
  292. `assets/x_grippers/<name>/` directory via a PR.
  293. ---
  294. ## 7. End-to-end: grasps → cuRobo motion planning → sim
  295. `end2end/` wires GraspGenX into a closed-loop pick-and-place pipeline:
  296. **GraspGenX** grasps per object → **cuRobo** plans a collision-free
  297. approach→grasp→lift trajectory to a reachable, collision-free grasp →
  298. **Newton/MuJoCo** replays it under gravity + contacts → renders **MP4** + exports
  299. **USD** for Isaac Sim / Omniverse.
  300. ### Install
  301. ```bash
  302. uv sync --extra end2end # cuRobo, Newton/MuJoCo, CoACD, USD, fcl
  303. python end2end/setup_end2end_deps.py # clones cuRobo source into ext/curobo, builds merged URDFs
  304. ```
  305. ### Run a demo (from repo root, headless)
  306. ```bash
  307. PYOPENGL_PLATFORM=egl PYGLET_HEADLESS=true uv run python end2end/e2e_grasp_demo.py \
  308. --robot_config end2end/robots/franka_panda.yaml \
  309. --env_config end2end/envs/single_bin_demo.yaml \
  310. --task clutter_pick_and_drop --playback_mode dynamic --no-viser \
  311. --num_grasps 200 --topk 80 --grasp_threshold 0.7 --planner graspmoe \
  312. --seed 0 --export-trajectory end2end/runs/franka_single/trajectory.json
  313. ```
  314. Three demos ship: Franka single-object → bin (above), Franka 3-object clutter →
  315. bin (`envs/franka_clutter3_demo.yaml`, add `--max_retries_per_object 2`), and
  316. UR10e + arx_x5 pick-and-lift (`robots/ur10e_arx_x5.yaml` +
  317. `envs/tabletop_single_nobin.yaml`, add `--mesh_file … --hold_after_close_frames
  318. 150`). Each writes a `trajectory.json`; render it decoupled from the sim:
  319. ```bash
  320. PYOPENGL_PLATFORM=egl uv run python end2end/render_trajectory_mp4.py \
  321. --trajectory end2end/runs/franka_single/trajectory.json \
  322. --output end2end/runs/franka_single/demo.mp4 --resolution 320x240 --no-texture
  323. PYOPENGL_PLATFORM=egl uv run python end2end/export_trajectory_usd.py \
  324. --trajectory end2end/runs/franka_single/trajectory.json \
  325. --output end2end/runs/franka_single/demo.usda
  326. ```
  327. ### How the grasps drive cuRobo
  328. The bridge between a GraspGenX grasp and a cuRobo plan is small and reusable:
  329. 1. **Frame convert.** A predicted grasp is a world-frame `(4,4)` pose in the
  330. *canonical grasp frame*. Each robot YAML supplies a `grasp_to_tool_transform`
  331. (grasp frame → the arm's tool0/TCP frame); apply it so cuRobo targets the
  332. tool link, not the grasp point. (See `robot_profiles.py`,
  333. `end2end/visualize_scene_grasps.py`.)
  334. 2. **Collision-filter first.** Score each grasp's *gripper mesh* against the
  335. table/bin/neighbors/target with fcl and **hand cuRobo only the collision-free
  336. grasps** — cuRobo never plans to a geometrically colliding grasp.
  337. 3. **Plan.** Build the goal with `curobo_compat.grasp_goals(...)` (returns a
  338. `curobo.types.Pose` or, on the lab fork, a `GoalToolPose`) and call
  339. `MotionPlanner.plan_grasp(...)`. Pass **all** top-K feasible grasps at once
  340. and cuRobo picks the most reachable; it returns approach→grasp→lift (falling
  341. back to approach+grasp when no feasible retraction exists). `curobo_compat.py`
  342. papers over the two `plan_grasp` signatures.
  343. Minimal shape of the handoff (see `e2e_grasp_demo.py` / `clutter_task.py`):
  344. ```python
  345. from curobo.motion_planner import MotionPlanner, MotionPlannerCfg
  346. from curobo_compat import grasp_goals
  347. # grasps_world: (K,4,4) from GraspGenX, already filtered to collision-free
  348. tool_poses = [world_grasp @ grasp_to_tool_transform for world_grasp in grasps_world]
  349. goal = grasp_goals(tool_poses, target_link=tool_frame) # Pose / GoalToolPose
  350. result = planner.plan_grasp(goal, start_state) # approach → grasp → lift
  351. ```
  352. ### ⚠️ Real-world requirements & limitations
  353. The demos use complete mesh-sampled or clean simulated point clouds. Deploying
  354. on a real robot adds several requirements, and each is a source of error that
  355. bounds grasp performance:
  356. - **Calibrated camera.** You need a camera with known **intrinsics** (focal
  357. length, principal point — for back-projecting depth to a metric point cloud)
  358. and **extrinsics** (the camera→robot-base transform — so planned grasps land in
  359. the robot's frame). `infer_scene_depth` takes intrinsics directly; extrinsics
  360. are what put the grasp where cuRobo can reach it.
  361. - **Good depth.** Grasp quality is bounded by depth quality; noisy or incomplete
  362. depth (reflective/transparent/thin objects, stereo holes) degrades grasps. Feed
  363. it good depth via **[FoundationStereo](https://github.com/NVlabs/FoundationStereo)**
  364. (learned stereo → dense accurate depth from a stereo pair) or a **LIDAR /
  365. time-of-flight** depth camera for direct high-quality range data.
  366. - **Instance segmentation of the target.** The model consumes a *segmented*
  367. object point cloud (or a per-object instance mask). Segment the target with an
  368. instance segmenter such as **SAM2 or SAM3**, then use the mask to carve the
  369. object's points out of the scene cloud — that segmented cloud is the input to
  370. `infer` / `infer_object` (or feed the mask to `infer_scene_depth` /
  371. `infer_scene_pc`).
  372. - **Downstream execution error.** Even with perfect grasps, **camera-calibration
  373. error** (intrinsics/extrinsics) and **robot control / execution error** (joint
  374. tracking, TCP calibration, gripper timing) shift where the gripper actually
  375. arrives versus where the grasp was predicted — both degrade real-world success
  376. independent of the model. Budget for them when a sim-perfect grasp misses on
  377. hardware.
  378. ---
  379. ## Debugging & gotchas
  380. - **Two-process order for MCP/serving:** start the **ZMQ server first**, then the
  381. MCP bridge or example client. Use `graspgenx_health_check` (MCP) or
  382. `--host/--port` to confirm reachability.
  383. - **Viewer:** demos are fixed to viser `:8080` (no `--port` flag); forward the
  384. port over SSH.
  385. - **Planner choice:** `graspmoe` (default) is highest quality; `diffusion` is
  386. faster.
  387. - **`grasp_threshold`:** `0.7` keeps confident grasps; `-1.0` disables the
  388. threshold and returns top-k instead.
  389. - **Tests:** `uv run --no-sync pytest -m "not end2end"` (fast);
  390. `uv run pytest tests/test_end2end_demos.py -m end2end -v -s` (slow, GPU).
README.md · README · 424 lines

View this file on GitHub

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <!-- <img src="fig/cover.png" width="1000" height="250" title="readme1"> -->
  2. <div align="center">
  3. <img src="assets/cover2.png" alt="GraspGenX logo" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
  4. <br>
  5. <h1>GraspGenX: Cross-Embodiment Foundation Model for Grasp Generation</h1>
  6. </div>
  7. <p align="center">
  8. <a href="https://graspgenx.github.io">
  9. <img alt="Project Page" src="https://img.shields.io/badge/Project-Page-F0529C">
  10. </a>
  11. <a href="https://arxiv.org/abs/2606.00998">
  12. <img alt="Arxiv paper link" src="https://img.shields.io/badge/arxiv-2606.00998-blue">
  13. </a>
  14. <a href="https://huggingface.co/adithyamurali/GraspGenXModel">
  15. <img alt="Model Checkpoints link" src="https://img.shields.io/badge/%F0%9F%A4%97%20HF-Models-yellow">
  16. </a>
  17. <a href="https://www.youtube.com/watch?v=a2sv9EVQJXE&feature=youtu.be">
  18. <img alt="Video link" src="https://img.shields.io/badge/video-red">
  19. </a>
  20. <a href="LICENSE">
  21. <img alt="GitHub License" src="https://img.shields.io/badge/License-Apache%202.0-76B900.svg">
  22. </a>
  23. </p>
  24. GraspGenX is a cross-embodiment grasp generation framework that produces high-quality 6-DOF grasps for **any** robot gripper — including novel grippers not seen during training. Unlike [GraspGen](https://github.com/NVlabs/GraspGen), which trains a separate model per gripper, GraspGenX trains a **single** model conditioned on a gripper representation derived from the gripper's swept volume. This enables generalization across grippers with different morphologies, kinematics, and degrees of freedom. We release pretrained model checkpoints trained with a large-scale simulated grasp dataset, spanning **over 2 billion grasps** computed across **32 procedurally generated grippers** in 6 kinematic families and **8000+ objects**.
  25. <!-- <img src="assets/gifs/clutter_removal.gif" width="180" height="135" title="readme1"> <img src="assets/gifs/real_g1.gif" width="90" height="135" title="readme2"> <img src="assets/gifs/real_piper.gif" width="171" height="135" title="readme3"> <img src="assets/gifs/surge.gif" width="153" height="135" title="readme4"> <img src="assets/gifs/battery_removal.gif" width="180" height="135" title="readme5"> -->
  26. <table>
  27. <tr>
  28. <td><img src="assets/gifs/clutter_removal.gif" height="135" alt="Clutter removal"></td>
  29. <td><img src="assets/gifs/real_g1.gif" height="135" alt="Unitree G1"></td>
  30. <td><img src="assets/gifs/real_piper.gif" height="135" alt="Real Piper"></td>
  31. <td><img src="assets/gifs/surge.gif" height="135" alt="Surge hand"></td>
  32. <td><img src="assets/gifs/battery_removal.gif" height="135" alt="Battery removal"></td>
  33. </tr>
  34. </table>
  35. ## Contents
  36. 1. [Installation](#installation)
  37. - [Docker](#installation-with-docker)
  38. - [uv Installation](#installation-with-uv)
  39. - [Fast-Inference Performance](#fast-inference-performance)
  40. 2. [Inference Demos](#inference-demos)
  41. - [Object Point Clouds](#predicting-grasps-for-segmented-object-point-clouds)
  42. - [Object Meshes](#predicting-grasps-for-object-meshes)
  43. - [Scene Point Clouds](#predicting-grasps-for-objects-from-scene-point-clouds)
  44. 3. [End-to-End Demo And VLA Data Generation](#end-to-end-demos)
  45. 4. [Integrating a New Gripper](#integrating-a-new-gripper)
  46. 5. [Agentic Workflows](#agentic-workflows)
  47. 6. [FAQ](#faq)
  48. 7. [License](#license)
  49. 8. [Citation](#citation)
  50. 9. [Contact](#contact)
  51. ## Installation
  52. Choose your preferred installation method. For training, we recommend **Docker**. For inference, **uv** is sufficient.
  53. | Method | Use Case | Complexity |
  54. |--------|----------|------------|
  55. | **Docker** | Training + Inference | ⭐⭐⭐ |
  56. | **uv** | Inference | ⭐ |
  57. ### Installation with uv
  58. Install [uv](https://docs.astral.sh/uv/getting-started/installation/). Eager
  59. PyTorch inference is the base install:
  60. ```bash
  61. git clone https://github.com/NVlabs/GraspGenX.git && cd GraspGenX
  62. uv sync
  63. ```
  64. Fast inference is a separate native C++ runtime. Build it on the target GPU:
  65. ```bash
  66. uv run --quiet --extra fast-inference graspgenx-install-fast-inference \
  67. --diffusion-grasps 100 \
  68. --obb-grasps 50
  69. ```
  70. The default encoder profile is `[3500,2048,1024,512,512]`. Use
  71. `--ptv3-capacities INPUT POOL1 POOL2 POOL3 POOL4` to build another profile.
  72. See [capacity profiles](cpp_runtime/README.md#configurable-ptv3-capacities)
  73. for supported limits and custom artifact directories.
  74. To install fast inference together with the Viser and rendering dependencies:
  75. ```bash
  76. uv sync --quiet --extra fast-inference --extra visualization
  77. uv run --quiet --extra fast-inference --extra visualization graspgenx-install-fast-inference \
  78. --diffusion-grasps 100 \
  79. --obb-grasps 50
  80. ```
  81. The installer command installs the Python build dependencies, downloads the
  82. released checkpoints when needed, exports GPU-specific TensorRT plans, builds
  83. the C++ runtime into the package search path, and runs an end-to-end smoke test.
  84. It reuses complete plans and incremental CMake output on subsequent runs. The
  85. installer shows only phase summaries by default and prints captured tool output
  86. if a command fails. Add `--verbose` to stream ONNX, TensorRT, and CMake output.
  87. TensorRT plan generation uses builder optimization level 2 by default for a
  88. reasonable install time. Add `--tensorrt-optimization-level 3` or
  89. `--tensorrt-optimization-level 5` to spend more build time searching for faster
  90. plans. Add `--force-artifacts` as well when rebuilding existing plans.
  91. The host still needs a CUDA toolkit with `nvcc` and matching TensorRT runtime
  92. and development headers (`libnvinfer` and `NvInfer.h`). For a nonstandard CUDA
  93. or TensorRT installation, pass `--cuda-root` or `--tensorrt-root`. On Jetson,
  94. the installer uses JetPack's system TensorRT Python package so plan generation
  95. matches the native TensorRT runtime exactly.
  96. Inside an existing conda or Python environment, use `pip install -e .` for
  97. PyTorch inference. Use the uv commands above for the complete native runtime;
  98. they select the CUDA 13 PyTorch and TensorRT packages together.
  99. Add `--extra visualization` only when running the Viser or rendering demos,
  100. `--extra usd` only for USD loading/export on supported platforms, and `--extra
  101. training` only for dataset and training utilities.
  102. See [Fast inference](FAST_INFERENCE.md) for the optimized pipeline and the
  103. [native C++ TensorRT runtime](cpp_runtime/README.md) for build and API details.
  104. ### Fast-Inference Performance
  105. Measurements use the native C++ runtime, the released 20-step DDPM schedule,
  106. 3,500 model input points, 100 diffusion candidates, and 50 analytic OBB
  107. candidates. Each result reports the mean of three independent process means;
  108. each process performs 300 CUDA Graph warmup replays followed by 300 timed
  109. replays. The timed boundary starts with a CUDA-resident point cloud and ends
  110. with ranked, CUDA-resident grasps and scores.
  111. | Target | 150 candidates (100 diffusion + 50 OBB) |
  112. |---|---:|
  113. | NVIDIA RTX 6000 Ada | 3.178 ms |
  114. Every benchmark output contains the exact requested count, finite poses and
  115. scores, descending confidence, deterministic replay, and no PTV3
  116. topology-capacity overflow.
  117. These measurements use TensorRT 10.16.1.11, the default capacity profile, and a
  118. synthetic object. They exclude initialization, plan construction, and input
  119. transfers. See the [benchmark guide](cpp_runtime/README.md#validation-and-benchmark)
  120. for commands and the [input limits](cpp_runtime/README.md#input-limits-and-concurrency).
  121. ### Installation with Docker
  122. Training support targets the Docker environment.
  123. ```bash
  124. git clone https://github.com/NVlabs/GraspGenX.git && cd GraspGenX && bash docker/build.sh
  125. ```
  126. ## Setup Checkpoints and Gripper Assets
  127. ### Checkpoints
  128. The GraspGenX model checkpoints download automatically on the first import from [`https://huggingface.co/adithyamurali/GraspGenXModel`](https://huggingface.co/adithyamurali/GraspGenXModel) and clone to `<repo_root>/ext/graspgenx_checkpoints`. If the checkpoints already exist elsewhere, specify the path with this environment variable:
  129. ```bash
  130. export GRASPGENX_CHECKPOINT_DIR=/path/to/your/graspgenx_checkpoints
  131. ```
  132. ### Gripper Config and Assets
  133. Each gripper needs a corresponding config file (with sweep volume information etc.) and URDF as an input to GraspGenX. We curate this gripper metadata for many popular grippers in [gripper_descriptions](https://huggingface.co/datasets/adithyamurali/gripper_descriptions). This package supports inference with those grippers and clones automatically on the first import to `<repo_root>/ext/gripper_descriptions`. If the package already exists elsewhere, specify its location with this environment variable:
  134. ```bash
  135. export GRASPGENX_GRIPPER_CFG_DIR=/path/to/your/gripper_descriptions
  136. ```
  137. Verify the assets can be imported properly:
  138. ```bash
  139. uv run python -c "from graspgenx import get_gripper_descriptions_root; print(get_gripper_descriptions_root())"
  140. ```
  141. To list all the available grippers:
  142. ```bash
  143. uv run python scripts/list_grippers.py
  144. ```
  145. ## Inference Demos
  146. GraspGenX supports two input modalities:
  147. 1. **Partial point cloud observations** — e.g. from a depth camera
  148. 2. **Object meshes** — `.obj`, `.stl`, `.ply`, or USD formats
  149. ### Predicting grasps for segmented object point clouds
  150. The demo script generates grasp poses for segmented object point clouds and visualizes the results using [viser](https://viser.studio/). You can access the GUI by visiting [http://localhost:8080](http://localhost:8080) on a browser. Note that exact port may be different sometimes (it will be specified in the log output). Click Next to cycle through the objects. The gripper mesh for the top scoring grasp is visualized in blue.
  151. ```bash
  152. uv run python scripts/demo_object_pc.py \
  153. --sample_data_dir assets/sample_data/real_world \
  154. --gripper_name robotiq_2f_85 \
  155. --plot_top_mesh
  156. ```
  157. Add `--fast-inference --num_grasps 100` to use the native C++ runtime with the
  158. 100-diffusion-candidate plans built above. Candidate counts must match the
  159. installed plans. Python passes its
  160. CUDA point-cloud pointer and current stream through a small C ABI; C++ owns the
  161. complete PTV3, TensorRT, diffusion, OBB, scoring, ranking, and CUDA Graph
  162. pipeline. Set
  163. `GRASPGENX_CPP_RUNTIME_LIBRARY` and `GRASPGENX_CPP_RUNTIME_ARTIFACTS` if the
  164. library and artifacts are outside their default `cpp_runtime` locations.
  165. <table>
  166. <tr>
  167. <td><img src="assets/figs/obj_pc_robotiq1.png" width="300" height="250" title="obj_pc_1"></td>
  168. <td><img src="assets/figs/obj_pc_robotiq2.png" width="268" height="250" title="obj_pc_2"></td>
  169. <td><img src="assets/figs/obj_pc_surge1.png" width="258" height="250" title="obj_pc_2"></td>
  170. <td><img src="assets/figs/obj_pc_unitree_g11.png" width="258" height="250" title="obj_pc_3"></td>
  171. </tr>
  172. </table>
  173. Replace `--gripper_name` with any supported gripper, e.g. `franka_umi`, `franka_panda`, `robotiq_2f_85`, `robotiq_2f_140`, `unitree_g1`, `inspire_hand`, `surge_hand`, `barrett_hand`, `galaxea_g1`, `ezgripper`. Run `uv run python scripts/list_grippers.py` for the available grippers. By default the demo uses the **GraspMoE** planner (grasps sampled with diffusion ∪ Oriented-Bounding grasps, all scored by the discriminator). To fall back to the diffusion-only planner: `--planner diffusion`. To restrict the GraspMoE OBB sweep to the top face only: `--moe_obb_density sparse` (single centroid pose) or `--moe_obb_density dense` (positions along the OBB's longer XY axis, top-down only).
  174. ### Predicting grasps for objects from scene point clouds
  175. The scene point cloud demo loads full-scene point clouds (with per-object segmentation), runs GraspGenX inference on every segmented object, and visualizes the predicted grasps in the context of the full scene. Grasps that would collide with the rest of the scene are filtered out. You can access the GUI by visiting [http://localhost:8080](http://localhost:8080) on a browser.
  176. ```bash
  177. uv run python scripts/demo_scene_pc.py \
  178. --sample_data_dir assets/sample_data/real_world \
  179. --scene 00 \
  180. --gripper_name robotiq_2f_85 \
  181. --planner diffusion \
  182. --num_grasps 100 \
  183. --fast-inference
  184. ```
  185. With `--fast-inference`, every segmented object uses the shared native C++ runtime.
  186. Scene objects are handled in a fast loop that reuses the same fixed one-object
  187. CUDA Graph, so arbitrary scene sizes do not create new runtime instances or
  188. TensorRT contexts. Scene `00` contains eight valid segmented objects after
  189. loader validation. The script prints total and amortized timing before adding
  190. the grasps to viser.
  191. <table cellspacing="20">
  192. <tr>
  193. <td style="padding: 0 15px;"><img src="assets/figs/scene_pc_1.png" height="300" width="420" title="scenepc1"></td>
  194. <td style="padding: 0 15px;"><img src="assets/figs/scene_pc_2.png" height="300" width="420" title="scenepc2"></td>
  195. <td style="padding: 0 15px;"><img src="assets/figs/scene_pc_3.png" height="300" width="300" title="scenepc3"></td>
  196. <td style="padding: 0 15px;"><img src="assets/figs/scene_pc_4.png" height="300" width="300" title="scenepc4"></td>
  197. </tr>
  198. </table>
  199. By default, the model runs GraspMoE (Grasp Mixture-of-Experts) sampling with both the diffusion model or Oriented-Bounding-Box (OBB)-based heuristics. To use just the diffusion model, you can use `--planner diffusion` and to remove collision checking, you can add `--no-filter_collisions`.
  200. ### Predicting grasps for object meshes
  201. The script samples points from the mesh surface, runs GraspGenX inference, and visualizes both the mesh and predicted grasps. You can access the GUI by visiting [http://localhost:8080](http://localhost:8080) on a browser.
  202. ```bash
  203. uv run python scripts/demo_object_mesh.py \
  204. --mesh_file assets/sample_data/object_mesh/banana.obj \
  205. --mesh_scale 1.0 \
  206. --gripper_name inspire_hand \
  207. --grasp_threshold -1.0 --return_topk --topk_num_grasps 100 --plot_top_mesh
  208. ```
  209. Supported mesh formats: `.obj`, `.stl`, `.ply`, and USD (`.usd`, `.usda`, `.usdc`, `.usdz`). USD files are loaded via [scene_synthesizer](https://github.com/NVlabs/scene_synthesizer). For headless/batch inference (no visualization), add `--no-visualization --output_file /tmp/grasps.yml`.
  210. <table>
  211. <tr>
  212. <td><img src="assets/figs/obj_mesh_arx_x5.png" width="300" title="arx_x5"></td>
  213. <td><img src="assets/figs/obj_mesh_surge_hand.png" width="268" title="arx_x5"></td>
  214. <td><img src="assets/figs/obj_mesh_unitree_g1.png" width="258" title="arx_x5"></td>
  215. </tr>
  216. </table>
  217. ## End-to-End Demos
  218. Beyond predicting grasps, [`end2end/`](end2end/README.md) wires GraspGenX into
  219. a full closed-loop pick-and-place pipeline: **GraspGenX** grasps → **cuRobo**
  220. collision-free motion planning → **Newton/MuJoCo** physics replay (gravity +
  221. contacts) → **MP4** render + **USD** export for IsaacSim / Omniverse. Three
  222. example demos ship out of the box — a Franka Panda (single object → bin, and a
  223. 3-object clutter scene → bin) and a UR10e with a multi-finger hand
  224. (arx_x5 pick-and-lift).
  225. This pipeline needs an extra dependency stack (cuRobo, Newton, MuJoCo, CoACD,
  226. USD, fcl):
  227. ```bash
  228. uv sync --extra end2end
  229. ```
  230. See [`end2end/README.md`](end2end/README.md) for the run command, config
  231. layout, and visualization tools.
  232. ## Integrating a New Gripper
  233. GraspGenX runs zero-shot on any gripper given a URDF + meshes. The only thing GraspGenX needs that a stock URDF does not provide is a `config.json` describing the open/close joint states and the swept-volume bounding boxes that condition the model. The interactive **gripper config wizard** generates that file (and copies the URDF/meshes into the right place) for you; everything else — visual mesh, collision mesh, point-cloud cache, etc. — is produced automatically downstream.
  234. ### Run the wizard
  235. ```bash
  236. uv run python scripts/gripper_config_wizard.py \
  237. --urdf /path/to/your/gripper.urdf \
  238. --name <gripper_name> \
  239. --port 8081
  240. ```
  241. For example, to onboard the right Sharpa hand from a sibling `gripper_descriptions` checkout:
  242. ```bash
  243. uv run python scripts/gripper_config_wizard.py \
  244. --urdf ../gripper_descriptions/gripper_descriptions/assets/sharpa/right_sharpa_wave.urdf \
  245. --name sharpa_right \
  246. --port 8081
  247. ```
  248. Open `http://localhost:8081` and step through the GUI. There are six panels — the wizard prompts you in order and you click "Confirm" to advance:
  249. 1. **Align Base Frame.** Rotate so `+Z` is the approach axis (along the fingers) and `+X` is the closing direction.
  250. 2. **Confirm Open Configuration.** Drive joint sliders to the fully-open pose.
  251. 3. **Annotate Open Sweep Volume.** Drag the blue box to enclose the inner finger volume at the open state.
  252. 4. **Annotate Half-Open Sweep Volume.** Set the closed pose, then drag the orange box to enclose the inner volume at the half-open state.
  253. 5. **Review Animation.** Watch open ↔ close with both boxes overlaid; go back if anything looks off.
  254. 6. **Confirm and Save.** Pick the gripper type (`parallel_2f`, `revolute_2f`, `revolute_3f`) and symmetry flag, then save.
  255. ### What gets written
  256. On save, the wizard writes a fresh directory at `<gripper_descriptions_root>/assets/x_grippers/<name>/` (resolved via `$GRASPGENX_GRIPPER_CFG_DIR`, or the auto-cloned `<repo>/ext/gripper_descriptions/` — see [Gripper Config and Assets](#gripper-config-and-assets)) containing:
  257. - `gripper.urdf` and `meshes/` — copied from your source URDF.
  258. - `config.json` — the open/close joint states, fingertip, swept-volume boxes, gripper type, and base rotation you annotated in the GUI. This is the file the model conditions on.
  259. - `vis_mesh.obj` — merged visual mesh of the gripper at the open pose.
  260. The collision mesh and point-cloud caches needed at inference time are generated lazily on first use; you don't need to run anything else.
  261. ### Verify
  262. Spin up the descriptor viewer to sanity-check the saved config (animates open ↔ close with both sweep volumes overlaid):
  263. ```bash
  264. uv run python scripts/vis_gripper_desc.py --gripper <gripper_name> --port 8081
  265. ```
  266. Then run inference against a point cloud as you would for any built-in gripper. Any of the demo scripts should work:
  267. ```bash
  268. uv run python scripts/demo_object_pc.py \
  269. --sample_data_dir assets/sample_data/real_world \
  270. --gripper_name <gripper_name> \
  271. --plot_top_mesh
  272. ```
  273. ### Please upstream your gripper
  274. If you've onboarded a gripper that isn't already in [`gripper_descriptions`](https://huggingface.co/datasets/adithyamurali/gripper_descriptions), please consider opening a PR with the saved `assets/x_grippers/<name>/` directory. The community benefits enormously from a growing library of curated grippers, and your config will let everyone else skip the wizard for that hand.
  275. ### Procedural Gripper Assets
  276. Procedural gripper meshes and URDFs are stored under `assets/proc_grippers/`, organized by kinematic family. Each gripper directory contains the URDF, collision meshes, and swept volume point cloud.
  277. ## Agentic Workflows
  278. See `mcp/` and `client-server/`.
  279. ## Testing
  280. The `end2end` takes a while longer and can be skipped:
  281. ```bash
  282. uv run --no-sync pytest -m "not end2end"
  283. ```
  284. ## FAQ
  285. ### How does GraspGenX differ from GraspGen?
  286. GraspGen trains a **separate model per gripper**. GraspGenX trains a **single model** that generalizes across grippers by conditioning on a swept-volume representation of the gripper. This means GraspGenX can generate grasps for novel grippers not seen during training, without re-training.
  287. ### What grippers does GraspGenX support?
  288. GraspGenX supports any gripper for which you can compute a swept volume. The released model's training set contains 32 procedural grippers spanning 6 kinematic families (parallel 2-finger, revolute 2-finger, revolute 3-finger — each with two design variants). It generalizes zero-shot to multiple real-world grippers.
  289. ### How do I add a new gripper?
  290. See the [Integrating a New Gripper](#integrating-a-new-gripper) section. The interactive wizard generates `config.json` (and copies the URDF/meshes) for any gripper given just its URDF.
  291. ### Can I use the GraspGen checkpoints with GraspGenX?
  292. No. GraspGen and GraspGenX use different model architectures (per-gripper vs. cross-embodiment conditioning). You need GraspGenX-specific checkpoints.
  293. ### How do I report a bug?
  294. Please post a GitHub issue and we will follow up, or email us directly.
  295. ## License
  296. This project is licensed under the [Apache License 2.0](LICENSE). The model checkpoints are released under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/).
  297. Please email us (admurali@nvidia.com) if you are using this repository for commerical deployment and require any support.
  298. ## Citation
  299. If you found this work useful, please consider citing:
  300. ```
  301. @inproceedings{graspgenx2026,
  302. title = {GraspGen-X: Cross-Embodiment 6-DOF Diffusion-based Grasping},
  303. author = {Han, Beining and Chao, Yu-Wei and Coumans, Erwin and Eppner, Clemens and Sundaralingam, Balakumar and Deng, Jia and Birchfield, Stan and Murali, Adithyavairavan},
  304. booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  305. year = {2026},
  306. }
  307. ```
  308. ## Third-Party Software
  309. This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.
  310. ## Contact
  311. Please reach out to [Adithya Murali](http://adithyamurali.com) (admurali@nvidia.com) for further enquiries.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.