Skip to content
Back to search

asset-harvester

NVIDIA Asset Harvester is a generative AI model for autonomous vehicle simulation that extracts reusable 3D Gaussian assets directly from captured driving scenes, even from partial or obstructed views.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/NVIDIA/asset-harvester with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/NVIDIA/asset-harvester'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/nvidia--asset-harvester/badge.svg)](/listings/nvidia--asset-harvester)

HTML

<a href="/listings/nvidia--asset-harvester"><img src="/api/listings/nvidia--asset-harvester/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Install benchmark evaluation dependencies and download third-party model weights required for evaluation. The script requires Hugging Face access to `facebook/dinov3-vith16plus-pretrain-lvd1689m` and `facebook/sam-3d-body-dinov3`.

Note that the benchmark environment needs `transformers>=4.56.0`, while the main asset-harvester environment uses `transformers==4.48.3`. Therefore, we clone the environment into a separate benchmark environment and install the updated dependencies.

conda create --name av-object-benchmark --clone asset-harvester

conda activate av-object-benchmark 
bash benchmark/install.sh

This installs benchmark-specific packages (`pytorch-lightning`, `yacs`, `ninja`, `termcolor`, `transformers>=4.56.0`), clones the SAM 3D Body repo, and downloads the SAM 3D Body and DINOv3 checkpoints. If the third-party checkpoints are unavailable (gated repo), eval still runs but skips embedding metrics.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 283 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Asset Harvester: Extracting 3D Assets from Autonomous Driving Logs for Simulation
  2. [![Project Page](https://img.shields.io/badge/Project-Page-green)](https://research.nvidia.com/labs/sil/projects/asset-harvester/)
  3. [![Paper](https://img.shields.io/badge/arXiv-Paper-b31b1b?logo=arxiv)](https://arxiv.org/pdf/2604.18468)
  4. [![License](https://img.shields.io/badge/License-Apache--2.0-orange)](LICENSE.txt)
  5. [![Model](https://img.shields.io/badge/HF-Models-yellow?logo=huggingface&style=flat-square)](https://huggingface.co/nvidia/asset-harvester)[![Live Demo!](https://img.shields.io/badge/Live%20Demo!-cd3233?logo=database&logoColor=white&style=flat-square)](https://huggingface.co/spaces/nvidia/asset-harvester)[![NCore Data](https://img.shields.io/badge/NCore-0d9488?logo=database&logoColor=white&style=flat-square)](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NCore)[![Benchmark](https://img.shields.io/badge/Benchmark-4f46e5?logoColor=white&style=flat-square)](https://huggingface.co/datasets/nvidia/NuRec-AV-Object-Benchmark)
  6. NVIDIA Asset Harvester is a generative AI model that extracts 3D Gaussian object representations directly from captured driving scenes, even from partial or obstructed views, removing manual asset creation. Each extracted asset feeds into Harmonizer for scene integration, enabling scalable data generation for autonomous vehicle simulation.
  7. ### Abstract
  8. Closed-loop simulation is a core component of autonomous vehicle (AV) development, enabling scalable testing, training, and safety validation before real-world deployment. Neural scene reconstruction converts driving logs into interactive 3D environments for simulation, but it does not produce complete 3D object assets required for agent manipulation and large-viewpoint novel-view synthesis.
  9. To address this challenge, we present Asset Harvester, an image-to-3D model and end-to-end pipeline that converts sparse, in-the-wild object observations from real driving logs into complete, simulation-ready assets.
  10. Rather than relying on a single model component, we developed a system-level design for real-world AV data that combines large-scale curation of object-centric training tuples, geometry-aware preprocessing across heterogeneous sensors, and a robust training recipe that couples sparse-view-conditioned multiview generation with 3D Gaussian lifting. Within this system, SparseViewDiT is explicitly designed to address limited-angle views and other real-world data challenges.
  11. Together with hybrid data curation, augmentation, and self-distillation, this system enables scalable conversion of sparse AV object observations into reusable 3D assets.
  12. <p align="center">
  13. <img src="docs/assets/teaser.gif" alt="Asset Harvester teaser" width="100%" style="border: none;">
  14. </p>
  15. **Asset Harvester** turns real-world driving logs into complete, simulation-ready 3D assets — from just one or a few in-the-wild object views. It handles vehicles, pedestrians, riders, and other road objects, even under heavy occlusion, noisy calibration, and extreme viewpoint bias. A multiview diffusion model generates consistent novel viewpoints, and a feed-forward Gaussian reconstructor lifts them to full 3D in seconds. The result: high-fidelity 3D Gaussian splat assets ready for insertion into simulation environments. The pipeline plugs directly into NVIDIA NCore and NuRec for scalable data ingestion and closed-loop simulation.
  16. ## Pipeline Overview
  17. NCore V4 Data ─► NCore Parsing ─► Multiview Diffusion + Gaussian Lifting ─► [`metadata.yaml`](docs/end_to_end_example.md#step-3-generate-external-assets-metadata-to-use-with-nvidia-omniverse-nurec-optional) (required for NuRec Object Insertion)
  18. <table>
  19. <tr>
  20. <th>Input View</th>
  21. <th>Multiview Diffusion (2 of 16 views shown) </th>
  22. <th>3D Gaussian Lifting </th>
  23. </tr>
  24. <tr>
  25. <td><img src="docs/assets/demo_input.jpeg" width="256"></td>
  26. <td><img src="docs/assets/demo_mv_concat.png" width="512"></td>
  27. <td><img src="docs/assets/demo_lifting.gif" width="256"></td>
  28. </tr>
  29. <tr>
  30. <td><img src="docs/assets/demo_input_truck.jpeg" width="256"></td>
  31. <td><img src="docs/assets/demo_mv_truck_concat.png" width="512"></td>
  32. <td><img src="docs/assets/demo_lifting_truck.gif" width="256"></td>
  33. </tr>
  34. </table>
  35. ## User Guide
  36. For end-to-end asset harvesting from recorded driving sessions, see our <b>[Full End-to-End Workflow](docs/end_to_end_example.md)</b> :sparkles: !
  37. <details>
  38. <summary><b>Setup</b></summary>
  39. #### Prerequisites
  40. - **conda** (Miniconda or Miniforge)
  41. - **NVIDIA driver** >= 570 (CUDA 12.8 compatible)
  42. - **GCC** 10–13 (tested with GCC 12.3)
  43. - **GPU VRAM** ~16 GB (add `--offload_model_to_cpu` to offload unused models to CPU for lower VRAM)
  44. > **Note:** Initial setup takes ~20 minutes to complete.
  45. ```bash
  46. git clone https://github.com/NVIDIA/asset-harvester.git
  47. cd asset-harvester
  48. bash setup.sh
  49. conda activate asset-harvester
  50. ```
  51. > **Option note:** `bash setup.sh --env-name asset-harvester --python 3.10`
  52. The bash script `setup.sh` handles the full environment setup for this repo.
  53. If you need a manual install from a checkout, preinstall the pinned `gsplat` build first, then install the repo with the extras you need:
  54. ```bash
  55. pip install --no-cache-dir --no-build-isolation \
  56. "git+https://github.com/nerfstudio-project/gsplat.git@b60e917c95afc449c5be33a634f1f457e116ff5e"
  57. pip install --extra-index-url https://download.pytorch.org/whl/cu128 \
  58. -e ".[ncore-parser,multiview_diffusion,tokengs,camera-estimator]"
  59. ```
  60. #### Download Model Checkpoints
  61. ```bash
  62. pip install huggingface_hub[cli]
  63. hf auth login
  64. hf download nvidia/asset-harvester --local-dir checkpoints
  65. ```
  66. or, manually from the [Hugging Face](https://huggingface.co/nvidia/asset-harvester).
  67. This places the following files in `checkpoints/`:
  68. ```
  69. checkpoints/
  70. ├── AH_multiview_diffusion.safetensors
  71. ├── AH_tokengs_lifting.safetensors
  72. ├── AH_camera_estimator.safetensors
  73. └── AH_object_seg_jit.pt
  74. ```
  75. </details>
  76. <details>
  77. <summary><b>Image-to-3D</b></summary>
  78. Try Asset Harvester on our sample data (Multiview Diffusion + Gaussian Lifting).
  79. Requires ~16GB VRAM. If you run into VRAM OOM issues, add `--offload_model_to_cpu` to offload unused model components to CPU:
  80. ```bash
  81. export DATA_ROOT=data_samples/rectified_AV_objects/
  82. export CHECKPOINT_MV=checkpoints/AH_multiview_diffusion.safetensors
  83. export CHECKPOINT_GS=checkpoints/AH_tokengs_lifting.safetensors
  84. export OUTPUT_DIR=outputs/harvesting
  85. python3 run_inference.py \
  86. --diffusion_checkpoint "${CHECKPOINT_MV}" \
  87. --data_root "${DATA_ROOT}" \
  88. --output_dir "${OUTPUT_DIR}" \
  89. --lifting_checkpoint "${CHECKPOINT_GS}"
  90. ```
  91. Or if you have a single-view image with an object in the center and a foreground mask, resize them into 512x512, and place them in a folder with this structure:
  92. ```
  93. YOUR_IMAGE_ROOT/
  94. ├─── YOUR_IMAGE_NAME_0
  95. │ ├── frame.jpeg
  96. │ └── mask.png
  97. └─── YOUR_IMAGE_NAME_1
  98. ...
  99. ```
  100. If masks are not available, you can also use our image segmentation model to
  101. get mask.png from frame.jpeg stored in above structure:
  102. ```bash
  103. export CHECKPOINT_SEG=checkpoints/AH_object_seg_jit.pt
  104. export IMAGE_ROOT=data_samples/segmented_images
  105. python -m asset_harvester.utils.image_segment \
  106. --checkpoint $CHECKPOINT_SEG \
  107. --image_folder $IMAGE_ROOT \
  108. --frame_name frame.jpeg \
  109. --mask_name mask.png
  110. ```
  111. Check the folder `data_samples/OOD_images` for example.
  112. After data preparation, run Asset Harvester with our built-in camera estimator:
  113. ```bash
  114. export YOUR_IMAGE_ROOT=data_samples/OOD_images
  115. export CHECKPOINT_MV=checkpoints/AH_multiview_diffusion.safetensors
  116. export CHECKPOINT_GS=checkpoints/AH_tokengs_lifting.safetensors
  117. export CHECKPOINT_CAM=checkpoints/AH_camera_estimator.safetensors
  118. export OUTPUT_DIR=outputs/harvesting_with_camera_estimate
  119. python3 run_inference.py \
  120. --diffusion_checkpoint "${CHECKPOINT_MV}" \
  121. --ahc_checkpoint "${CHECKPOINT_CAM}" \
  122. --image_dir "${YOUR_IMAGE_ROOT}" \
  123. --output_dir "${OUTPUT_DIR}" \
  124. --lifting_checkpoint "${CHECKPOINT_GS}"
  125. ```
  126. </details>
  127. <details>
  128. <summary><b>Full End-to-End Workflow</b></summary>
  129. For the complete step-by-step pipeline walkthrough — from raw NCore driving logs through NCore parsing, multiview diffusion, Gaussian lifting, metadata generation, and benchmark evaluation — see the **[End-to-End Guide](docs/end_to_end_example.md)**.
  130. </details>
  131. <details>
  132. <summary><b>Post-Training (Fine-Tuning)</b></summary>
  133. The SparseViewDiT multiview diffusion model can be fine-tuned on your own parsed samples with the same rectified-flow objective it was pretrained with, built on HuggingFace `diffusers` + `accelerate` (full fine-tuning or LoRA, single- or multi-GPU). See the **[Post-Training Guide](docs/post_training.md)**.
  134. </details>
  135. <a id="benchmark"></a>
  136. <details>
  137. <summary><b>Benchmark</b></summary>
  138. Evaluate `run_inference.py` output by rendering Gaussian splat assets from reserved-view cameras and comparing to ground-truth frames. Computes **PSNR**, **LPIPS**, **SSIM**, and optional **DINOv3 embedding** distances.
  139. #### Setup
  140. Install benchmark evaluation dependencies and download third-party model weights required for evaluation. The script requires Hugging Face access to `facebook/dinov3-vith16plus-pretrain-lvd1689m` and `facebook/sam-3d-body-dinov3`.
  141. Note that the benchmark environment needs `transformers>=4.56.0`, while the main asset-harvester environment uses `transformers==4.48.3`. Therefore, we clone the environment into a separate benchmark environment and install the updated dependencies.
  142. ```bash
  143. conda create --name av-object-benchmark --clone asset-harvester
  144. conda activate av-object-benchmark
  145. bash benchmark/install.sh
  146. ```
  147. This installs benchmark-specific packages (`pytorch-lightning`, `yacs`, `ninja`, `termcolor`, `transformers>=4.56.0`), clones the SAM 3D Body repo, and downloads the SAM 3D Body and DINOv3 checkpoints. If the third-party checkpoints are unavailable (gated repo), eval still runs but skips embedding metrics.
  148. #### Usage
  149. ```bash
  150. python benchmark/eval.py --output_dir /path/to/run_inference_output --eval_output_dir benchmark/eval
  151. ```
  152. See [`benchmark/README.md`](benchmark/README.md) for the full list of options, input/output format, and metric descriptions.
  153. </details>
  154. <details>
  155. <summary><b>Repository Structure</b></summary>
  156. ```
  157. asset-harvester/
  158. ├── asset_harvester/
  159. │ ├── camera_estimator/ # Camera pose estimator package
  160. │ ├── multiview_diffusion/ # SparseViewDiT package
  161. │ ├── ncore_parser/ # NCore parser package + CLI
  162. │ ├── patches/ # Compatibility patches
  163. │ ├── tokengs/ # TokenGS package + main.py training entry point
  164. │ └── utils/ # Shared utilities and CLI helpers
  165. ├── benchmark/ # Evaluation tools
  166. │ ├── eval.py
  167. │ ├── embedding_metrics.py
  168. │ ├── utils.py
  169. │ ├── install.sh
  170. │ └── README.md
  171. ├── data_samples/ # Bundled sample data for Quick Start
  172. ├── docs/
  173. │ ├── assets/ # Demo images, GIFs, and videos
  174. │ ├── end_to_end_example.md
  175. │ └── tokengs.md
  176. ├── scripts/
  177. │ ├── check_license_headers.py
  178. │ └── run_ncore_parser.sh # Step 1 wrapper for the parser CLI
  179. ├── run_inference.py # Main inference entry point
  180. ├── run.sh # Step 2: multiview diffusion + Gaussian lifting wrapper
  181. ├── setup.sh
  182. ├── pyproject.toml
  183. ├── CONTRIBUTING.MD
  184. ├── LICENSE.txt
  185. ├── THIRD_PARTY_LICENSE.txt
  186. └── README.md
  187. ```
  188. </details>
  189. ## License
  190. This project is licensed under the Apache License 2.0. See individual file headers for details.
  191. ## Support
  192. 📣 **Asset Harvester usage questions and discussion**: please post on the [NVIDIA Developer Forum (Omniverse / NuRec)](https://forums.developer.nvidia.com/c/omniverse/platform/nurec/752).
  193. 🐛 **Asset Harvester code-level bugs, documentation issues, and feature requests**: file a [GitHub issue](../../issues/new/choose) using the appropriate template (Bug report, Documentation request, or Feature request). The relevant NVIDIA responder is auto-assigned via the `assignees:` field on the template.
  194. 🚨 **Security vulnerabilities**: please use [NVIDIA's Vulnerability Disclosure Program](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail). Do not file security issues publicly here.
  195. ## Citation
  196. If you find this work useful in your research, please consider citing:
  197. ```bibtex
  198. @article{cao2026assetharvester,
  199. title = {Asset Harvester: Extracting 3D Assets from Autonomous Driving Logs for Simulation},
  200. author = {Cao, Tianshi and Ren, Jiawei and Zhang, Yuxuan and
  201. Seo, Jaewoo and Huang, Jiahui and Solanki, Shikhar and
  202. Zhang, Haotian and Guo, Mingfei and Turki, Haithem and
  203. Li, Muxingzi and Zhu, Yue and Zhang, Sipeng and Gojcic, Zan and
  204. Fidler, Sanja and Yin, Kangxue},
  205. year = {2026},
  206. }
  207. ```
  208. ## Disclaimer
  209. Asset Harvester is trained for the AV domain, and the results are not guaranteed in other domains.
  210. AI models generate responses and outputs based on complex algorithms and machine learning techniques, and those responses or outputs may be inaccurate or offensive. By downloading a model, you assume the risk of any harm caused by any response or output of the model. By using this software or model, you are agreeing to the terms and conditions of the license, acceptable use policy, and privacy policy as applicable.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.