Nvidia's new world model helps robots navigate the world

Nvidia unveiled Cosmos 3, an open AI world model designed to help robots, autonomous vehicles and other physical systems better understand and predict real-world environments. Why it matters: Nvidia is continuing its move beyond chips into AI models and software, positioning itself to become a foundational platform for physical AI development. Driving the news: Nvidia says it trained Cosmos 3 on 20 trillion tokens of multimodal data, including nearly a billion images, 400 million real and synthetic videos, ambient audio, text and action data from humans and robots. That action data is what makes Cosmos different from a regular video generator. It's meant to model how machines move, not just how scenes look, Ming-Yu Liu, VP of Nvidia's Cosmos Lab, told Axios. Autonomous actions are key. Developers can use Cosmos 3 to simulate actions in physical environments, then build task-specific models for robots and other machines on top of it. Cosmos 3 is designed to generate action data — such as robot joint angles, gripper positions and trajectories — that can help train machines to navigate and manipulate the physical world. Between the lines: Cosmos is an open model, similar to its early Nemotron family, making it easier for hardware makers to customize Cosmos to their needs and ensure that future versions more closely align to the needs of the industry, Liu said. Nvidia is also establishing a coalition of companies supporting the effort. Initial partners include Agile Robots , Black Forest Labs and Runway. Nvidia says Cosmos can generate rare or dangerous scenarios — such as robot collisions or unusual road events — that are difficult, expensive or unsafe to capture repeatedly. Zoom in: Nvidia is releasing two versions immediately: a "super" model for tasks requiring high physics accuracy, such as training robots and autonomous vehicles, and a "nano" model that can generate results in fractions of a second. An "edge" model that can run loc...

Original source: Axios