NVIDIA Cosmos for Developers
NVIDIA Cosmos™ is a platform of state-of-the-art generative world foundation models (WFMs), advanced tokenizers, guardrails, and an accelerated data processing and curation pipeline for autonomous vehicles (AVs) and robotics developers.
Build, evaluate, deploy, and simulate physical AI models faster while minimizing testing and validation risks in the real world.
NVIDIA Cosmos World Foundation Models
A family of pre-trained models for world generation as videos for accelerating physical AI development. Available openly to developers on NGC, Hugging Face, and GitHub.
Cosmos Predict
For out-of-the box world generation and post-training.
A generalist model that generates world states from text or video prompts and synthesizes continuous motion by predicting frames between a given start and end frame.
These models range from 4 billion to 15 billion parameters and can be used based on inference requirements.
Cosmos Transfer
For controllable and photoreal synthetic data at scale.
Input: Segmentation maps, depth signals, LiDAR scans, key points, trajectories, HD maps and ground truth simulations from NVIDIA Omniverse.
Output: Photorealistic world scenes, conditioned based on inputs, mirroring layout, object placement and motion.
Cosmos Reason
For physical AI reasoning.
Fully customizable, multimodal reasoning model trained using visual-language fine-tuning and reinforcement learning that uses a chain of thoughts to plan responses.
The model enables intelligent decision-making by reasoning and rewarding optimal responses.
Currently in early access
Cosmos WFM Post-Training Samples
Post-trained Cosmos Predict WFM generates predictive world states for autonomous vehicles, creating single or multi-view videos from ground truth input for 360° environmental awareness in AV training.
Workflow Enablers
Prompt Upsampler: Transform original input prompts into more detailed and enriched versions for higher-quality outputs.
Guardrails: Set of guardrails including a pre-guard to block harmful inputs and a post-guard to ensure safety and consistency in generations.
Cosmos Tokenizers
A suite of image and video tokenizers that advances the state-of-the-art in visual tokenization for world model training.
Introductory Resources
Scale Synthetic Data and Physical AI Reasoning With Cosmos WFMs
New models and controls released at GTC for generating use-case-specific synthetic data at scale further accelerate WFM development.
Enhanced Transfer and Predict Capabilities for Cosmos WFMs
Cosmos Transfer WFMs for photorealistic video output from structured inputs and Cosmos Predict WFMs for multi-frame generation are accelerating physical AI development.
Introducing Cosmos for Physical AI Development
Get an introduction to the models, tools, and capabilities of the Cosmos platform to accelerate the development of physical AI-embodied systems such as robots and autonomous vehicles.
Starter Kits
Start solving physical AI challenges by developing custom world models with Cosmos or using Cosmos WFMs for downstream use cases. Explore implementation scripts, explainer blogs, and more how-to documentation for various stages of physical AI development.
For Synthetic Data Generation
Build and deploy world models for infinite domain-specific synthetic data. Use NVIDIA Omniverse for physics-based conditioning.
For Post-Training Cosmos WFMs
Cosmos WFMs are purpose-built for post-training. Use domain-specific datasets to build world models or post-train for different types of output like action generation for policy models.
Cosmos Learning Library
More Resources
Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
NVIDIA has collaborated with Google Deepmind to watermark generated videos from the NVIDIA API catalog.
For more detailed information on ethical considerations for this model, please see the System Card, Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards. Please report security vulnerabilities or NVIDIA AI concerns here.
Get Started With NVIDIA Cosmos Today