Skip to content
Back to search

omni-dreams

NVIDIA OmniDreams is a world model that generates photorealistic video for autonomous-driving simulation in real time.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/nv-tlabs--omni-dreams/badge.svg)](/listings/nv-tlabs--omni-dreams)

HTML

<a href="/listings/nv-tlabs--omni-dreams"><img src="/api/listings/nv-tlabs--omni-dreams/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

For fine-tuning the Cosmos2 SV-HDMap world model on a single 8-GPU node (student-init, bidirectional teacher, or self-forcing distillation), see [`samples/post-training/QUICKSTART.md`](samples/post-training/QUICKSTART.md). It covers HuggingFace auth setup, checkpoint + sample-dataset staging, and the launcher invocations for all three experiments. Slurm wrapper included for multi-tenant clusters.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 131 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # NVIDIA OmniDreams
  2. **Resources:** [Research blog](https://research.nvidia.com/labs/sil/projects/omnidreams-blog/) | [Technical Report](https://arxiv.org/abs/2606.03159) | [Model weights](https://huggingface.co/nvidia/omni-dreams-models)
  3. NVIDIA OmniDreams is a world model that generates multi-camera photorealistic
  4. video for autonomous-driving simulation in real time.
  5. The model consumes:
  6. - A single input frame
  7. - Initial text prompt
  8. - Per-frame coarse HD map image and trajectory poses
  9. It produces photorealistic video frames in chunks.
  10. ## How the pieces fit together
  11. Rollout starts from one real RGB frame. That frame anchors the
  12. appearance of the scene. The text prompt describes the driving context, while
  13. the per-frame HD map image and trajectory poses provide the structured
  14. conditioning for each generated chunk. The world model autoregressively
  15. generates the next video chunk, then feeds that chunk back into the next step
  16. so the scene can continue over time.
  17. Interactive inference and live driving demos now live in the companion
  18. [`flashdreams`](https://github.com/NVIDIA/flashdreams) project. This repo owns
  19. the post-training sample and release tree.
  20. This repository contains the following samples for demonstration:
  21. - **[`samples/post-training`](samples/post-training/README.md)** - launchers
  22. and configs for fine-tuning the Cosmos2 SV-HDMap world model on a single
  23. 8-GPU node or a Slurm cluster (student-init, bidirectional teacher, and
  24. self-forcing distillation). See
  25. [`samples/post-training/QUICKSTART.md`](samples/post-training/QUICKSTART.md)
  26. for the four-step out-of-box flow.
  27. For interactive driving and offline, reproducible batch `mp4` video
  28. generation, see the companion [`flashdreams`](https://github.com/NVIDIA/flashdreams)
  29. project.
  30. ## Community
  31. Join us on the NVIDIA Omniverse Discord to share your results, attend office
  32. hours, and take part in technical discussion with the NVIDIA OmniDreams team and
  33. community.
  34. If you are new, start with the
  35. [server invite](https://discord.com/invite/nvidiaomniverse) to on-board.
  36. Channels:
  37. - [`#omnidreams`](https://discord.gg/cruUPk9tuS)
  38. - [`#flashdreams`](https://discord.gg/cMt2mHm4aN)
  39. - [`#world-models-chit-chat`](https://discord.gg/AkcK7NVGQD)
  40. ## Prerequisites
  41. ### Hardware and disk
  42. | Workflow | Tested / expected hardware | Disk guidance |
  43. |---|---|---|
  44. | Post-training | Supported minimum is a single 8-GPU Ampere/Hopper node (`NPROC=8`). Smaller `NPROC` values are unsupported. | At least 150 GB free; 200 GB or more is recommended for caches plus training output. |
  45. | Inference / interactive driving | Use the FlashDreams OmniDreams runner. | Follow the FlashDreams model and sample docs for cache and asset sizing. |
  46. Use a recent NVIDIA driver compatible with the CUDA stack in the selected
  47. workflow. The post-training quickstart was validated with driver 570.148.08
  48. and CUDA 12.8 on 8x H100 80 GB HBM3.
  49. ### Hugging Face access
  50. The post-training sample stages both checkpoints and gated Hugging Face sample
  51. data on first run. The sample dataset repo and branch are:
  52. - [`nvidia/PhysicalAI-Autonomous-Vehicles-NuRec`](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NuRec/tree/26.01) — PAI-NuRec 26.01 sample scenes for post-training
  53. You also need access to the OmniDreams model checkpoint repo under your
  54. authorized Hugging Face org. By default the post-training scripts look under
  55. `nvidia`; set `OMNI_DREAMS_HF_ORG=<YOUR-HF-ORG>` if your access lives under a
  56. different org.
  57. To allow automated download, create a Hugging Face token at https://huggingface.co/settings/tokens/new.
  58. ```bash
  59. export HF_TOKEN=<YOUR-HF-TOKEN> # optional: used by quick access checks only
  60. ```
  61. The post-training runtime reads the token from
  62. `$OMNI_CACHE_DIR/huggingface/token`, not directly from `HF_TOKEN`. Follow the
  63. post-training quickstart's staging step to write the token file before running
  64. `setup_env.sh`.
  65. If any download fails with `401`, `403`, or a gated-repo error, verify both the
  66. token and the repo access above before debugging anything else.
  67. ## Inference And Interactive Driving
  68. Use the FlashDreams OmniDreams runner for inference and interactive driving.
  69. The inputs are the same pieces described above: an initial RGB frame, a text
  70. prompt, and HD-map / trajectory conditioning frames. Batch inference produces
  71. a reproducible `mp4` sequence; the interactive-drive integration provides the
  72. live driving experience.
  73. The interactive driving sample that previously lived in this repository has
  74. moved to FlashDreams:
  75. - [FlashDreams OmniDreams model docs](https://nvidia.github.io/flashdreams/main/models/omnidreams.html)
  76. - [FlashDreams interactive-drive sample](https://github.com/NVIDIA/flashdreams/tree/main/integrations/omnidreams/omnidreams/interactive_drive)
  77. ## Quickstart: post-training (fine-tune)
  78. For fine-tuning the Cosmos2 SV-HDMap world model on a single 8-GPU node
  79. (student-init, bidirectional teacher, or self-forcing distillation), see
  80. [`samples/post-training/QUICKSTART.md`](samples/post-training/QUICKSTART.md).
  81. It covers HuggingFace auth setup, checkpoint + sample-dataset staging, and
  82. the launcher invocations for all three experiments. Slurm wrapper included
  83. for multi-tenant clusters.
  84. # License and Attribution
  85. OmniDreams is licensed under the [Apache License, Version 2.0](LICENSE).
  86. Third-party runtime dependencies are fetched by package managers or companion
  87. projects such as `flashdreams`.
  88. - [`LICENSE`](LICENSE) — repository-wide Apache-2.0 grant
  89. - [`CONTRIBUTING.md`](CONTRIBUTING.md) — DCO sign-off and PR conventions
  90. - [`REUSE.toml`](REUSE.toml) — per-path / per-file license metadata
  91. - [`THIRD_PARTY_NOTICES.txt`](./THIRD_PARTY_NOTICES.txt) — upstream
  92. attribution for vendored third-party code
  93. - [`NOTICE`](./NOTICE) — third-party-fetch notice for runtime / install-time
  94. downloads

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.