# Cosmos-Predict2 **Repository Path**: mirrors/Cosmos-Predict2 ## Basic Information - **Project Name**: Cosmos-Predict2 - **Description**: Cosmos-Predict2 是英伟达开源物理AI世界基础模型（WFMs）生态中的关键分支，专注于通过高级世界建模进行未来状态预测 - **Primary Language**: Python - **License**: Apache-2.0 - **Default Branch**: main - **Homepage**: https://www.oschina.net/p/cosmos-predict2 - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 0 - **Created**: 2025-06-13 - **Last Updated**: 2025-06-15 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README

NVIDIA Cosmos Header

### Paper (coming soon!) | [Website](https://research.nvidia.com/labs/dir/cosmos-predict2/) | [Hugging Face](https://huggingface.co/collections/nvidia/cosmos-predict2-68028efc052239369a0f2959) Cosmos-Predict2 is a key branch of the [Cosmos World Foundation Models](https://www.nvidia.com/en-us/ai/cosmos) (WFMs) ecosystem for Physical AI, specializing in future state prediction through advanced world modeling. It offers two powerful capabilities: text-to-image generation for creating high-quality images from text descriptions, and video-to-world generation for producing visual simulations from video inputs. We visualize the architecture of Cosmos-Predict2 in the following figure.

Cosmos-Predict Architecture Diagram

## Models * [Cosmos-Predict2-2B-Text2Image](https://huggingface.co/nvidia/Cosmos-Predict2-2B-Text2Image): Text-to-image generation * [Cosmos-Predict2-14B-Text2Image](https://huggingface.co/nvidia/Cosmos-Predict2-14B-Text2Image): Text-to-image generation * [Cosmos-Predict2-2B-Video2World](https://huggingface.co/nvidia/Cosmos-Predict2-2B-Video2World): Video + Text based future visual world generation * [Cosmos-Predict2-14B-Video2World](https://huggingface.co/nvidia/Cosmos-Predict2-14B-Video2World): Video + Text based future visual world generation * [Cosmos-Predict2-14B-Sample-GR00T-Dreams-GR1](https://huggingface.co/nvidia/Cosmos-Predict2-14B-Sample-GR00T-Dreams-GR1): Video + Text based future visual world generation, post-trained on GR00T Dreams GR1 dataset * [Cosmos-Predict2-14B-Sample-GR00T-Dreams-DROID](https://huggingface.co/nvidia/Cosmos-Predict2-14B-Sample-GR00T-Dreams-DROID): Video + Text based future visual world generation, post-trained on GR00T Dreams DROID dataset * [Cosmos-Predict2-2B-Sample-Action-Conditioned](https://huggingface.co/nvidia/Cosmos-Predict2-2B-Sample-Action-Conditioned): Video + Action based future visual world generation, post-trained on Bridge dataset --- ## Quick Start Here is a quick example demonstrating how to use Cosmos-Predict2-2B-Video2World for video generation: ```python import torch from imaginaire.utils.io import save_image_or_video from cosmos_predict2.configs.base.config_video2world import PREDICT2_VIDEO2WORLD_PIPELINE_2B from cosmos_predict2.pipelines.video2world import Video2WorldPipeline # Create the video generation pipeline. pipe = Video2WorldPipeline.from_config( config=PREDICT2_VIDEO2WORLD_PIPELINE_2B, dit_path="checkpoints/nvidia/Cosmos-Predict2-2B-Video2World/model-720p-16fps.pt", text_encoder_path="checkpoints/google-t5/t5-11b", ) # Specify the input image path and text prompt. image_path = "assets/video2world/example_input.jpg" prompt = "A high-definition video captures the precision of robotic welding in an industrial setting. The first frame showcases a robotic arm, equipped with a welding torch, positioned over a large metal structure. The welding process is in full swing, with bright sparks and intense light illuminating the scene, creating a vivid display of blue and white hues. A significant amount of smoke billows around the welding area, partially obscuring the view but emphasizing the heat and activity. The background reveals parts of the workshop environment, including a ventilation system and various pieces of machinery, indicating a busy and functional industrial workspace. As the video progresses, the robotic arm maintains its steady position, continuing the welding process and moving to its left. The welding torch consistently emits sparks and light, and the smoke continues to rise, diffusing slightly as it moves upward. The metal surface beneath the torch shows ongoing signs of heating and melting. The scene retains its industrial ambiance, with the welding sparks and smoke dominating the visual field, underscoring the ongoing nature of the welding operation." # Run the video generation pipeline. video = pipe(input_path=image_path, prompt=prompt) # Save the resulting output video. save_image_or_video(video, "output/test.mp4", fps=16) ``` **Input prompt:** > A high-definition video captures the precision of robotic welding in an industrial setting. The first frame showcases a robotic arm, equipped with a welding torch, positioned over a large metal structure. The welding process is in full swing, with bright sparks and intense light illuminating the scene, creating a vivid display of blue and white hues. A significant amount of smoke billows around the welding area, partially obscuring the view but emphasizing the heat and activity. The background reveals parts of the workshop environment, including a ventilation system and various pieces of machinery, indicating a busy and functional industrial workspace. As the video progresses, the robotic arm maintains its steady position, continuing the welding process and moving to its left. The welding torch consistently emits sparks and light, and the smoke continues to rise, diffusing slightly as it moves upward. The metal surface beneath the torch shows ongoing signs of heating and melting. The scene retains its industrial ambiance, with the welding sparks and smoke dominating the visual field, underscoring the ongoing nature of the welding operation. | Input image | Output video | |-------------|--------------| | ![Input Image](assets/video2world/example_input.jpg) | | --- ## User Guide Our [setup guide](documentations/setup.md) provides complete information on * [System requirements](documentations/setup.md#system-requirements): Detailed hardware and software prerequisites * [Installation](documentations/setup.md#installation): Step-by-step setup with both Conda and Docker options * [Downloading checkpoints](documentations/setup.md#downloading-checkpoints): Instructions for obtaining model weights * [Troubleshooting](documentations/setup.md#troubleshooting): Solutions for common installation and CUDA compatibility issues For inference examples and usage * **[Text2Image Inference](documentations/inference_text2image.md)**: Guide for generating high-quality images from text prompts * **[Video2World Inference](documentations/inference_video2world.md)**: Guide for generating videos from images/videos with text prompts, including: * Single and batch processing * Multi-frame conditioning * Multi-GPU inference for faster generation * Using the prompt refiner * Rejection sampling for quality improvement * **[Text2World Inference](documentations/inference_text2world.md)**: Guide for generating videos directly from text prompts, including: * Single and batch processing * Multi-GPU inference for faster generation For post-training customization * **[Post-training guide](documentations/post-training_video2world.md)**: General guide to the training system in the codebase * **[Post-training on Cosmos-NeMo-Assets](documentations/post-training_video2world_cosmos_nemo_assets.md)**: Case study for post-training on Cosmos-NeMo-Assets data * **[Post-training on fisheye-view AgiBotWorld-Alpha dataset](documentations/post-training_video2world_agibot_fisheye.md)**: Case study for post-training on fisheye-view robot videos from AgiBotWorld-Alpha dataset. * **[Post-training on GR00T Dreams GR1 and DROID datasets](documentations/post-training_video2world_gr00t.md)**: Case study for post-training on GR00T Dreams GR1 and DROID datasets. * **[Action-conditioned Post-training on Bridge dataset](documentations/post-training_video2world_action.md)**: Case study for action-conditioned post-training on Bridge dataset. Our [performance guide](documentations/performance.md) includes * [Hardware requirements](documentations/performance.md#hardware-requirements): Recommended GPU configurations and memory requirements * [Performance benchmarks](documentations/performance.md#performance-benchmarks): Detailed speed and quality comparisons across different GPU architectures * [Model selection guide](documentations/performance.md#model-selection-guide): Practical advice for choosing between 2B and 14B variants based on your needs --- ## Contributing We thrive on community collaboration! [NVIDIA-Cosmos](https://github.com/nvidia-cosmos/) wouldn't be where it is without contributions from developers like you. Check out our [Contributing Guide](CONTRIBUTING.md) to get started, and share your feedback through issues. Big thanks 🙏 to everyone helping us push the boundaries of open-source physical AI! --- ## License and Contact This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use. This model includes safety and content moderation features powered by Llama Guard 3. Llama Guard 3 is used solely as a content input filter and is subject to its own license. NVIDIA Cosmos source code is released under the [Apache 2 License](https://www.apache.org/licenses/LICENSE-2.0). NVIDIA Cosmos models are released under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license). For a custom license, please contact [cosmos-license@nvidia.com](mailto:cosmos-license@nvidia.com).