NVIDIA has announced Cosmos 3, a new world foundation model for physical AI such as robots and autonomous vehicles[1]. Unveiled at NVIDIA GTC Taipei at COMPUTEX, the model handles text, video, images, ambient sound, and action in a single model, and can generate training data grounded in real-world physical context[1].

What Is Cosmos 3 — A World Foundation Model That Even Generates Action Data

The real world is always in motion. To operate autonomously, physical AI systems such as robots, autonomous vehicles (AVs), and smart spaces need to understand not only what they see and what caused it, but also what is likely to happen next[1]. In a warehouse, a robot may encounter object configurations it has never seen; on the road, an AV may face a pedestrian stepping out from between parked cars; in a factory, a safety system must predict where a forklift is heading, not just detect that it is there[1]. Capturing and recreating such scenarios in the real world is slow, costly, and often impossible to repeat at scale[1].

Cosmos 3 is built for this loop[1]. It is a generalist foundation model trained on diverse data, giving it a broad understanding of how scenes, motion, and robotic actions relate[1]. It is also an omnimodel with native action generation, meaning it can directly output numerical action data such as joint angles, gripper positions, and trajectory points[1]. Robots need more than images or video of a scene to learn; for a pick-and-place task, for example, they need the action signals that guide how to reach, grasp, move, and place an object[1]. Developers can fine-tune Cosmos 3 to specialize their robots for a particular embodiment, camera layout, workspace, or task[1].

A Two-Stage Design: A Reasoning Block and a Generation Block

Cosmos 3 uses a mixture-of-transformers architecture[1]. A reasoning block first interprets what is happening in a scene, and a generation block then takes that context to produce physically grounded outputs[1]. These outputs range from synthetic video to robot-task data[1].

In other words, Cosmos 3 supports perception, prediction, and action in a single model[1]. It can reason about which objects are moving in a scene, where paths may intersect, and what state is likely to follow, and then generate dense captions, predicted scene changes, or scenario variations[1].

Use in Robotics, Autonomous Driving, and Smart Cities

NVIDIA's GEAR team is using Cosmos 3 to develop video action models[1]. The effort helps embodied agents learn to reason, move, and act across games, simulations, and real-world robotics environments[1].

Agile Robots, which builds humanoids and other embodiments, aims to handle industrial tasks autonomously, precisely, and efficiently with platforms such as Thor 3 and FR3[1]. The company uses Cosmos 3 to generate action-conditioned robot data and create diverse task trajectories at scale for its policy development[1]. According to NVIDIA, a Cosmos 3 Nano post-trained policy leads on RoboLab, which tests language-guided tasks in simulation, and on RoboArena, which compares policies on DROID robots in real-world environments[1].

Linker Vision, which builds smart-city and industrial solutions, uses Cosmos' vision language reasoning capabilities[1]. It analyzes thousands of live camera feeds in real time, understands spatial context, extracts useful insights, and performs root-cause analysis[1]. NVIDIA says Cosmos 3 is the top-ranked open vision language model on VANTAGE-Bench, which tests smart-infrastructure scene understanding, and on the TAR challenge, which tests traffic anomaly reasoning[1].

Collisions and rare, long-tail edge cases are among the most important examples for preparing humanoids, arm robots, and even surgical robots for the real world, yet they are hard to capture safely, repeatedly, and at scale[1]. As a video foundation model, Cosmos 3 can generate physically plausible video sequences to teach how the real world changes over time[1].

Benchmarks and Availability — Released as Open Models

According to NVIDIA, Cosmos 3 variants rank first on open-weights leaderboards on Artificial Analysis, and also top several world-generation benchmarks including Physics-IQ, R-Bench, and PAI-Bench[1].

Developers can try Cosmos 3 on build.nvidia.com and download open models from Hugging Face[1]. They can customize models and generate synthetic data using resources on GitHub, and deploy with NVIDIA NIM microservices[1]. The model uses the OpenMDW 1.1 license from the Linux Foundation, letting developers train, modify, redistribute, and deploy materials — including weights, architecture, documentation, datasets, benchmarks, and code — under a single, model-centric license[1]. The announcement accompanied a GTC Taipei keynote by NVIDIA founder and CEO Jensen Huang[1].

Summary

Cosmos 3 is a world foundation model for physical AI that brings perception, prediction, and action together and can even generate action data[1]. Its approach of answering the shared shortage of training data for "AI that acts on the real world" — robotics, autonomous driving, and smart cities — through synthetic data and future-state prediction is what makes it notable. Released as open models with a clear path to try it right away, it stands out as a move to broaden physical AI development.

Source: https://blogs.nvidia.com/blog/cosmos-3-physical-ai-open-world-foundation-model/