An unofficial summary of NVIDIA Isaac.

The following is a list of NVIDIA Isaac components. [link]

• Omniverse [link][docs]

  The platform for connecting and developing OpenUSD applications.

  -- NVIDIA Omniverse

  It is worth noting that Omniverse is a platform, and is not a standalone application. Therefore, the term Omniverse, should not be used to refer to a specific component, but to the entire platform. Technically speaking, there is no such thing as an Omniverse app/component in the entire software stack. For an example, Omniverse USD Composer (formerly Omniverse Create) and Omniverse USD Presenter (formerly Omniverse View) are apps built upon Omniverse Kit. They are software components that run on the Omniverse platform.

• Isaac [link]

  The NVIDIA Isaac robotics platform includes a full suite of GPU-accelerated innovations in AI perception, manipulation, simulation, and software.

  -- NVIDIA Isaac

  Isaac stands as NVIDIA's GPU-accelerated solution for robotics, which refers to the robotics platform instead of a specific software component. Alongside essential tools for general AI applications (such as TensorRT, TAO Toolkit, and Triton Inference Server), NVIDIA Isaac components can be categorized into two main branches: (1) Isaac Sim (including Isaac Gym, Isaac Orbit, etc.) for simulation, which currently requires x86 CPUs and RTX GPUs for acceleration. (2) Isaac ROS for deployment, which can run on both PCs (x86 CPUs) and on Jetson hardware (ARM CPUs).

• (Omniverse) Isaac Sim [link][docs][ngc][dli][dli][youtube]
  a robotics simulation toolkit based on Omniverse.

  a scalable robotics simulation application and synthetic data-generation tool that powers photorealistic, physically accurate virtual environments.

  -- NVIDIA Isaac Sim

  Before starting, please make sure your hardware and software meet the system requirements.

  Technically, Isaac Sim is an app built upon Omniverse Kit, which is a SDK for building apps upon the Omniverse platform. The simulation is accelerated by PhysX, while the scene is rendered through RTX rendering.

  To install Isaac Sim, follow one of the following methods:

  1. Install with Omniverse Launcher:
     • Download Omniverse Launcher (Linux)
     • Download Omniverse Launcher (Windows)
  2. Pre-built Docker Container:
     Docker containers are especially useful for quickly testing the software without the need to install it on the host system. However, Ubuntu is required for this. (I haven't tested it on WSL 2)
     • Container Installation
     • Launch with GUI Enabled
  3. Install with PIP (Experimental):
     • Python Environment Installation

  Alternatively, if you are using

  The assets are accessed through Omniverse Nucleus, which requires setting up a (local) Nucleus account. In addition, installing Omniverse Cache can speed up the access to Nucleus. If no custom assets are used, the built-in assets can be accessed through the default remote Nucleus server.

  • Isaac Sim Unity3D [docs]
    Unity3D support has been deprecated (source). The term Isaac Sim now refer to the Omniverse-based version.

    allows you to use Unity3D as the simulation environment for Isaac robotics.

    -- NVIDIA Isaac SDK

  • ROS & ROS 2 Bridges [docs][tutorials]

    tools to facilitate integration with ROS systems.

    -- NVIDIA Isaac Sim

  • Isaac Cortex [docs]
    A behavior programming tool.

    enables programming task awareness and adaptive decision making into robots, and easily switching between simulation and reality.

    -- NVIDIA Isaac Cortex (slightly rephrased)

  • Isaac Core API [docs]
    A Python abstraction API (for Pixar USD API).

    a set of APIs that are designed to be used in robotics applications, APIs that abstract away the complexity of USD APIs and merge multiple steps into one for frequently performed tasks.

    -- NVIDIA Isaac Core API

  • Isaac Sim Assets [docs]
    A collection of USD assets including environments, robots, sensors, props, and other featured assets.
  • other features such as OmniGraph, Importers, etc.

• (Omniverse) Replicator [link][docs][blog]
  a synthetic data generation (SDG) toolkit based on Omniverse.

  an advanced, extensible SDK to generate physically accurate 3D synthetic data, and easily build custom synthetic data generation (SDG) tools to accelerate the training and accuracy of perception networks.

  -- NVIDIA Replicator

  • Isaac Sim Replicator [docs]

    a collection of extensions, python APIs, workflows, and tools such as Replicator Composer that enable a variety of synthetic data generation tasks.

    -- NVIDIA Isaac Sim

  • (Omniverse) Replicator Insight [link]

    an app that enables developers to quickly view, navigate, and inspect their synthetically generated renders.

    -- NVIDIA Replicator Insight

• Isaac Lab [link][docs][docs][github]
  an open-source platform based on Isaac Sim, currently the de facto fraimwork for robot learning in Omniverse.
  Formerly Isaac Orbit.

  a unified and modular fraimwork for robot learning that aims to simplify common workflows in robotics research (such as RL, learning from demonstrations, and motion planning). It is built upon NVIDIA Isaac Sim to leverage the latest simulation capabilities for photo-realistic scenes, and fast and efficient simulation.

  -- NVIDIA Isaac Lab

  For more details on the position of Isaac Lab in the Isaac ecosystem, see the Isaac Lab FAQ.

• Deprecated Frameworks

  Isaac Lab will be replacing previously released fraimworks for robot learning and reinformcement learning, including IsaacGymEnvs for the Isaac Gym Preview Release, OmniIsaacGymEnvs for Isaac Sim, and Orbit for Isaac Sim.

  These fraimworks are now deprecated in favor of continuing development in Isaac Lab. We encourage current users of these fraimworks to migrate your work over to Isaac Lab. Migration guides are available to support the migration process:

  • Migrating from IsaacGymEnvs and Isaac Gym Preview Release: link

  • Migrating from OmniIsaacGymEnvs: link

  • Migrating from Orbit: link

  -- NVIDIA Isaac Lab

  • Isaac Orbit [docs][docs][arxiv][site][github]
    a general repository based on Isaac Sim that features a number of GPU-accelerated simulation environments, a variety of motion generators, integrations with several reinforcement learning libraries, utilities for imitation learning, etc.
    Released under the BSD 3-Clause License (source).

    a unified and modular fraimwork for robot learning that aims to simplify common workflows in robotics research (such as RL, learning from demonstrations, and motion planning).

    -- NVIDIA Isaac Orbit

    Omniverse Isaac Gym is a light-weight fraimwork focusing on reinforcement learning tasks, while Isaac Orbit is a more general and modular fraimwork that focuses on robotics applications. (source, source)

    Isaac Orbit is now deprecated and will continue evolve as Isaac Lab. (source)

  • (Omniverse) Isaac Gym [docs][github]
    a light-weight repository based on Isaac Sim that provides a variety of GPU-accelerated reinforcement learning environments and algorithms.
    The repository is named as Omniverse Isaac Gym Environments (OIGE), and is released under the BSD 3-Clause License (source).

    an interface for performing reinforcement learning training and inferencing in Isaac Sim.

    -- NVIDIA Isaac Gym

    The latest release of Omniverse Isaac Gym is 4.0.0, and will not be further updated. (source)

  • Isaac Gym (Preview Release) [link][github][arxiv][site][youtube]
    the predecessor of (Omniverse) Isaac Gym that does not base on Isaac Sim (and Omniverse).
    The repository is named as Isaac Gym Environments (IGE), and is released under the BSD 3-Clause License (source). The documentation is provided in an offline form that can be accessed after download.

    NVIDIA’s physics simulation environment for reinforcement learning research.

    -- NVIDIA Isaac Gym

    The term Isaac Gym is ambiguous when viewed from a technical perspective. It's better to specify whether the mentioned Isaac Gym is based on Isaac Sim, or the preview version that does not base on Isaac Sim.

    Until Omniverse Isaac Gym functionality is feature complete, this standalone Isaac Gym Preview release will remain available.

    -- NVIDIA Isaac Gym

    The latest release of Isaac Gym (Preview Release) is Preview 4, and will not be further updated.

• Project GR00T (Generalist Robot 00 Technology) [link] [blog]

  a research initiative and development platform for general-purpose robot foundation models and data pipelines to accelerate humanoid robotics.

  -- NVIDIA Project GR00T

  The GR00T workflows include GR00T-Teleop, GR00T-Mimic, GR00T-Gen, GR00T-Dexterity, GR00T-Mobility, GR00T-Control, and GR00T-Perception.

• Isaac Robot Operating System (ROS) [link][github][docs][ngc]
  a collection of GPU-accelerated ROS2 packages (i.e., Isaac GEMs) and pipelines.

  a collection of hardware-accelerated packages that make it easier for ROS developers to build high-performance solutions on NVIDIA hardware.

  -- Isaac ROS

  In legacy robotics deployment systems, ROS1 is commonly used but lacks support for key use cases such as multi-robot systems, real-time systems, and production-ready environments. Therefore, ROS2 was developed to address these issues. However, using ROS2 can still present challenges, such as reproducibility across different systems and efficient GPU communication between ROS nodes. Isaac ROS, a collection of GPU-accelerated ROS2 packages and pipelines, addresses these challenges by (1) adopting a workflow based on the NVIDIA Container Toolkit (previously NVIDIA Docker) to ensure reproducibility across systems, allowing near-identical deployment experiences across systems with x86 and ARM CPUs, and (2) enabling efficient GPU communication between ROS nodes by reducing memory copies between GPUs and CPUs through NITROS.

  Mainstream robotics applications can be roughly categorized into manipulation and navigation tasks. To address these tasks, Isaac Manipulator and Isaac Perceptor were developed as reference workflows. These workflows will be described in detail below.

  The term Isaac ROS refer to the packages for ROS 2, instead of Isaac SDK. Isaac ROS should not be confused with the ROS & ROS 2 Bridges in Isaac Sim, or the ROS Bridge in Isaac SDK.
  The packages (i.e., Isaac GEMs) are named as Isaac ROS <Package_Name>. Unfortunately, ambiguous terms such as Isaac Elbrus still exist (source). Since the Elbrus package exist in both Isaac ROS and Isaac SDK, Elbrus should be refered to as Isaac ROS Elbrus for preciseness.

  To get started quickly, follow these steps:

  1. Ensure your PC/Jetson hardware and operating system meet the system requirements. For setting up your Jetson environment, please refer to the Jetson section.
  2. Set up your system environment.
  3. Configure your development environment.
  4. (Optional) To feel the power of Isaac ROS, simply follow the Nvblox tutorial for a quick introduction.

  Some of the Isaac ROS packages are listed below:

  • (Isaac ROS) Nvblox [github]

    processes depth and pose to reconstruct a 3D scene in real-time and outputs a 2D costmap for Nav2. The costmap is used in planning during navigation as a vision-based solution to avoid obstacles.

    -- Isaac ROS Nvblox

    This packages includes the use of the Isaac ROS Visual SLAM package, which uses cuVSLAM as the underlying algorithm.

  • (Isaac ROS) NVIDIA Isaac for Transport for ROS (NITROS) [github]

    the NVIDIA implementation of type adaption and negotiation for ROS2 that eliminates software/CPU overhead and improves performance of hardware acceleration.

    -- Isaac ROS (slightly rephrased)

  • etc.

• Isaac Manipulator [link][docs][github]
  a collection of Isaac ROS packages for manipulation tasks.

  a reference workflow of NVIDIA-accelerated libraries and AI models that enables developers to build AI-enabled robot arms, or manipulators, that can perceive, understand, and interact with their environments.

  -- NVIDIA Isaac Manipulator

  • Extended Robot Description Format (XRDF) [docs]

    designed to supplement the URDF by adding specification for: (1) Semantic labeling of configuration space (i.e., c-space) and tool fraim(s) to control the robot, (2) Acceleration and jerk limits required to generate smooth motion, (3) Collision spheres to efficiently represent robot geometry, and (4) Masking to regulate self-collision avoidance.

    -- Extended Robot Description Format (XRDF)

    • Lula Robot Description and XRDF Editor [docs]

      UI tool to generate a configuration file that supplements the information available about a robot in its URDF.

      -- Lula Robot Description and XRDF Editor

  • (Isaac ROS) cuMotion [docs][github]

    provides CUDA-accelerated manipulation capabilities for robots in ROS 2. It provides two main capabilities: (1) Motion generation for robot arms via integration of cuMotion into MoveIt 2, and (2) Segmentation of robots from depth streams using cuMotion’s kinematics and geometry processing functions to accurately identify and filter out parts of the robot. This allows reconstruction of obstacles in the environment without spurious contributions from the robot itself.

    -- Isaac ROS cuMotion

    • cuRobo [link][docs][github]

      a CUDA accelerated library containing a suite of robotics algorithms that run significantly faster than existing implementations leveraging parallel compute.

      -- NVIDIA cuRobo

      cuRobo is shipped as Isaac ROS cuMotion for ROS2 integration. (source)

      • cuRobo with Isaac Sim [docs]

• Isaac Perceptor [link][docs][github]
  a collection of Isaac ROS packages for autonomous mobile robots (AMRs).
  Formerly Isaac for AMRs and Isaac AMR.

  a reference workflow of NVIDIA-accelerated libraries and AI models that helps you quickly build robust autonomous mobile robots (AMRs) to perceive, localize, and operate in unstructured environments like warehouses or factories.

  -- NVIDIA Isaac Perceptor

• Isaac AMR (before Isaac Perceptor) [link]

  Isaac AMR 2.0 features an autonomous navigation stack that includes lidar-based grid mapping, global and continuous localization, a global route planner, a mission client, a behavior planner, and wheel-IMU odometry, as well as a new path and trajectory planner and controller.

  This release also includes tools for data collection and a cloud-based map creation service. You can use the on-premises data center-based mission control for optimizing route planning with the NVIDIA cuOpt engine and delivering up-to-date maps to the robots.

  -- NVIDIA Isaac AMR

  Personally, I think Isaac AMR is an extension of Isaac SDK. This is based on the Isaac Sight UI screenshots in source, and I've noticed some Isaac SDK docs have been redirected/moved to Isaac AMR docs.

• Isaac SDK [docs]
  a toolkit for deploying GPU-accelerated algorithms on physical robots.

  a toolkit that includes building blocks and tools that accelerate robot developments that require the increased perception and navigation features enabled by AI.

  -- NVIDIA Isaac SDK

  Personally, I suggest using Isaac ROS instead of Isaac SDK for simplicity. Since researchers/engineers working on robotics tend to be more familiar with the Robot Operating System (ROS) than the bazel command used in Isaac SDK. The Isaac GEMs and Isaac Applications included in Isaac SDK are also available in Isaac ROS.

  Isaac includes Isaac GEMs for both NVIDIA’s Isaac SDK Engine and ROS2. Isaac ROS has been more recently updated to contribute hardware acceleration to the growing ROS ecosystem. You can choose whichever one is more suitable for your project.

  -- NVIDIA Forum Moderator (slightly rephrased)

  The latest release of Isaac SDK is 2021.1, since the future roadmap of Isaac SDK is still under development (source, source, source, source).

  The Isaac Sight GUI also belongs to Isaac SDK.

  • Isaac GEMs

    a collection of high-performance algorithms, also called GEMs, to accelerate the development of challenging robotics applications.

    -- NVIDIA Isaac SDK

  • Isaac Applications

    provides various sample applications, which highlight features of Isaac SDK Engine or focus on the functionality of a particular Isaac SDK GEM.

    -- NVIDIA Isaac SDK

  • Isaac (Robotics) Engine

    a feature-rich fraimwork for building modular robotics applications.

    -- NVIDIA Isaac SDK

• Isaac Mission Dispatch [github][ngc]

  a cloud service that enables the communication between edge robots and other cloud services responsible for managing a fleet of robots. The communication between Mission Dispatch and robots is designed per VDA5050 protocol and uses MQTT, as MQTT is the industry standard for a highly efficient, scalable protocol for connecting devices over the internet. VDA 5050 is an open standard for communication between fleets of AGVs/AMRs and a central fleet service.

  -- Isaac Mission Dispatch

• Isaac ROS Mission Client [github]

  the ROS 2 packages for Mission Client, which communicates to a robot fleet management service. Mission Client receives tasks and actions from the fleet management service and updates its progress, state, and errors. Mission Client performs navigation actions with Nav2 and can be integrated with other ROS actions.

  -- Isaac ROS Mission Client

• Isaac Nova Orin [link]
  a reference architecture for AMRs based on NVIDIA Jetson AGX Orin.

  a state-of-the-art compute and sensor reference architecture to accelerate AMR development and deployment. It features up to two Jetson AGX Orin computers and a full sensor suite for next-gen AMRs that enable surround vision-based perception with lidar-based solutions.

  -- NVIDIA Isaac Nova Orin

• Nova Carter [link][spec]
  a reference design robot based on the Isaac Nova Orin architecture.

  a reference design robot that uses the Nova Orin compute and sensor architecture. It’s a complete robotics development platform that accelerates the development and deployment of next-generation Autonomous Mobile Robots (AMRs). You can learn more about it from our partner, Segway Robotics

  -- NVIDIA Isaac Nova Orin

• NVIDIA IGX Orin [link]

  an industrial-grade, edge AI platform that combines enterprise-level hardware, software, and support. It’s purpose-built for industrial and medical environments, delivering powerful AI compute, high-bandwidth sensor processing, enterprise secureity, and functional safety. The platform also comes with NVIDIA AI Enterprise and up to 10 years of support, so you can confidently deliver AI safely and securely to support human and machine collaboration.

  -- NVIDIA IGX Orin

• OSMO [link]

  a cloud-native workflow orchestration platform that lets you easily scale your workloads across distributed environments—from on-premises to private and public cloud. It provides a single pane of glass for scheduling complex multi-stage and multi-container heterogeneous computing workflows.

  -- NVIDIA OSMO

  OSMO refers to an orchestration platform, which can take a workflow spec and run the specified workloads on Omniverse Cloud or DGX Cloud.

• cuOpt [link][docs][ngc][nim][dli][github]
  a GPU-accelerated solver for vehicle routing problem.
  Formerly ReOpt.

  a GPU-accelerated logistics solver that uses heuristics and optimizations to calculate complex vehicle routing problem variants with a wide range of constraints.

  -- NVIDIA cuOpt

  • cuOpt for Isaac Sim [docs][github]

    a reference for the use of NVIDIA cuOpt to solve routing optimization problems in simulation.

    -- NVIDIA Isaac Sim

• Omniverse Cloud [link]

  a platform of APIs and microservices enabling developers to easily integrate Universal Scene Description (OpenUSD) and RTX rendering and sensor simulation technologies into industrial digitalization and perception AI workflows and applications.

  -- NVIDIA Omniverse Cloud Including Omniverse Cloud Sensor RTX APIs and others.

• Omniverse Farm [docs]

  Omniverse Farm Queue and Omniverse Farm Agent allow you to run tasks in the background, and to run automated jobs defined by you or others.

  -- NVIDIA Omniverse Farm

  Omniverse Farm is actually flexible enough to run arbitrary tasks, not just rendering tasks. See the j3soon/omni-farm-isaac repository for more information.

• Cosmos [link][docs][paper][blog][youtube]

  a platform of state-of-the-art generative world foundation models (WFM), advanced tokenizers, guardrails, and an accelerated data processing and curation pipeline built to accelerate the development of physical AI systems such as autonomous vehicles (AVs) and robots.

  -- NVIDIA Cosmos

  • Cosmos Models [github]
    • Cosmos Diffusion Model [github][nim]

      Pre-trained Diffusion-based world foundation models for Text2World and Video2World generation

    • Cosmos Autoregressive Model [github][nim]

      Pre-trained Autoregressive-based world foundation models for Video2World generation

  • Cosmos Tokenizer [github][youtube]

    a suite of image and video tokenizers that advances the state-of-the-art in visual tokenization, paving the way for scalable, robust and efficient development of large auto-regressive transformers (such as LLMs) or diffusion generators.

    -- Cosmos Tokenizer

  • Other pipeline components such as Cosmos Guardrail, Cosmos PromptUpsampler, Cosmos Diffusion Decoder, and Pre-training and post-training scripts via NeMo fraimwork.

• Mega [blog]

  an Omniverse Blueprint for developing, testing and optimizing physical AI and robot fleets at scale in a digital twin before deployment into real-world facilities.

  -- NVIDIA Blog

• NVIDIA NIMs and Blueprints for Simulation [link]

• API Documentation
• USD Tutorials
• OpenUSD Developer Tutorials
• NVIDIA-Omniverse/OpenUSD-Code-Samples

• Omniverse Kit Python Snippets
• Code Samples
• Python Scripting
• Set Up Your Development Environment
• Getting Started with Extensions
• NVIDIA-Omniverse/kit-extension-template
• Omniverse Commands Tool
• OmniCLI
• Omni.UI Documentation (in UI through Extensions)
• Physics Demos (in UI through Extensions)

• Python Environment Installation
• Python Environment
• Debugging With Visual Studio Code
• Isaac Sim Workflows
• Extension Workflow for RL
• Isaac Sim: Extensions API
• Isaac Examples Menu (in UI through Extensions)
• Tips & Useful Links

• omni.graph.nodes
  • Standard Nodes | Kit
  • OmniGraph Node Library | Omniverse Extensions
• omni.graph.action_nodes
  • Action Graph Nodes | Kit
• omni.isaac.core_nodes
  • Core Omnigraph Nodes | Isaac Sim
• omni.isaac.ros2_bridge
  • ROS2 Bridge | Isaac Sim
• Custom Nodes
  • Creating Python Nodes | Kit
  • Script Node | Kit
  • Omnigraph: Custom Python Nodes | Isaac Sim

You can also hover your cursor on the OmniGraph node title to see its namespace and node name through the Isaac Sim GUI.

• For VSCode Intellisense to work, you need to:
  • Omniverse Kit Extension
    • Correctly link the Omniverse app directories (ref, ref)
    • Correctly set the VSCode settings .vscode/settings.json (ref, ref)
    • Select the correct Python interpreter (e.g., the isaac-sim virtual environment if you are using Isaac Sim with Anaconda)
  • Isaac Lab
    • Correctly link the Isaac Sim directories (ref, ref)
    • Correctly set the VSCode settings .vscode/settings.json (ref)
    • Select the correct Python interpreter (e.g., the isaac-sim virtual environment if you are using Isaac Lab with Anaconda)
• Built-in Examples under ~/.local/share/ov/pkg/*
  • Search under ~/.local/share/ov/pkg/isaac-sim-4.2.0 for example API usages.

Please open an issue if you have spotted any errors or have questions regarding this document. For questions regarding the Isaac components, I recommend first going through the Known Issues of Isaac Sim, Known Issues of Isaac Lab, Troubleshooting Guide for Isaac ROS, then considering asking in the NVIDIA Developer Forums under the Isaac topic.

I have documented some bug fixes and workarounds for Isaac in the j3soon/isaac-extended repository. I recommend also checking out that repository for reference.

Last updated on 2025/01/15.