Blog · Field notes · Oct 21, 2026 · 5 min
Physical AI now runs on open physics engines, world foundation models and synthetic demonstrations measured in hundreds of thousands. The newest benchmarks ask whether simulated scores predict real ones, and the part of the scene least checked against reality is the people.
Physical AI has a data problem that text AI never had. There is no web-sized archive of robots folding laundry, crossing a busy station or handing a box to a person. In 2025 the industry's answer became clear: build the missing data in simulation, from open physics engines, world foundation models and generated demonstrations, and mix it with what real robots record. The tools are now strong enough that the harder question has moved. It is how well a score in simulation predicts a score in the world, and the part of most scenes least checked against reality is the people.
NVIDIA set out the pieces at CES in January 2025 with Cosmos, a platform of world foundation models for robots and vehicles. Its stated uses are generating photoreal, physics-based synthetic data, searching video for rare scenarios such as snowy roads or congested warehouses, and testing policies in simulated situations, with Omniverse used to simulate possible outcomes.
At GTC in March 2025 NVIDIA released Isaac GR00T N1, an open foundation model for humanoid robots, with a concrete figure for what synthetic data can add. NVIDIA generated 780,000 synthetic trajectories in 11 hours, which it equates to about 6,500 hours of human demonstration, and reports that combining them with real data improved the model's performance by 40% over real data alone. The same announcement introduced Newton, a physics engine developed with Google DeepMind and Disney Research. In September 2025 Newton moved to the Linux Foundation as "an open source, GPU-accelerated, extensible physics engine" built on NVIDIA Warp and OpenUSD, so the physics underneath robot learning now has neutral governance.
Meta's V-JEPA 2 showed the other end of the scale: pre-trained on more than a million hours of internet video and then on under 62 hours of robot video, it planned pick-and-place actions zero-shot on robot arms in two labs where no data had been collected. The pattern across all three is the same. Large amounts of generated or general data, and a small amount of real data, go further together than either alone.
Simulation is also how robot policies are evaluated, and here 2025 brought a sober finding. The authors of PolaRiS, posted in December 2025, state that the visual and physical gap between existing simulation benchmarks and the real world "has made them an unreliable signal for policy improvement". Their remedy is to build simulated scenes from short video scans of real ones, and in paired tests they report that scores in those scenes correlate much more strongly with real-world performance than scores on existing benchmarks.
That is the right question to ask of any simulator, and it applies well beyond manipulation. A simulated score is worth what it predicts. The only way to know that is to compare the two on the same tasks, after the fact, without adjusting the simulator to fit.
Most of the effort in physical AI simulation has gone into the robot and its objects: contact, friction, deformable materials, lighting. The people in the scene usually receive far less. A warehouse, a hospital corridor, a station concourse or a city street is full of people who walk in groups, slow down, step around each other and react to the machine among them. In simulation they are usually moved by rule-based crowd models, and the social-force crowd is common in robot training.
Whole-city simulation is arriving too. NVIDIA's Omniverse Blueprint for smart city AI, announced in June 2025, builds simulation-ready twins of cities from aerial, satellite and map data, and uses them to train and test vision agents. As these scenes become the places where machines are prepared for the world, the realism of the people in them matters as much as the realism of the buildings.
The question for a simulated crowd is the same one PolaRiS asks of a simulated kitchen: compared with recordings of real people that the model never saw, does it move like them? Walking speeds, spacing, path straightness and how often people come within touching distance are all measurable, and public recordings of real pedestrians exist to measure them against.
A robot that has only met simulated people who keep perfect distance from each other will meet real ones who walk four abreast, stop suddenly and step close to see what it is carrying. If the simulated crowd is wrong in a consistent way, every measure of safety and comfort computed in simulation inherits the error. Checking the people is as much a part of sim-to-real as checking the friction.
Living Worlds simulates crowds and the people around machines, and scores them against recordings of real pedestrians that no run had opened.
Its first registered crowd beat the social-force crowd common in robot training, and people who ignore each other, in only 3 of 6 public ETH and UCY scenes where 5 were needed: in the dense scenes real people walk in groups and the crowd kept everyone apart (Not met). In four Jülich experiments it then won 1 of 4 (Not met). The next crowd moved more like real people than social force and people who ignore each other in more than half the runs of four fresh Jülich experiments (Met), and among real shoppers in the ATC shopping centre in Osaka it won all six three-minute windows of a day nobody had opened (Met, 6 of 6). A learned layer on how people walk then came closer to real walking than both, in at least two thirds of the runs of every set (Met); that study lists what it does not show, including counterflow, crossings and bottlenecks.
With a robot placed in the crowd on its recorded path, in a room where visitors carried boxes, buckets and large objects, Living Worlds predicted how close real people came to the real robot more accurately than a crowd blind to the robot and people who ignore it, in 7 of 8 recordings on two days nobody had opened (Met). Each study was registered before the data was read and scored once.
That is the ground robot training needs, and it is one of the uses Living Worlds is built for. Robots practise their tasks in a twin of the place they will work, thousands of times before they meet the real one, among simulated people who move the way real people did. The crowds are shown; robots trained among them come next.
Sources
Blog
Living Worlds' crowd against real pedestrians in Jülich experiments, a shopping centre in Osaka and six recorded street scenes, each scored once on recordings nobody had opened: met in the shopping centre, in four fresh Jülich experiments and with a learned walking layer; not met in the first Jülich set and in three of six street scenes.
An agent that remembers what a person said predicted their answers four years later better than a persona, in every panel and on every question; an agent given only a life record did no better than a persona. And Living Worlds' crowd predicted how close real people came to a real robot in 7 of 8 recordings nobody had opened.
World models now generate explorable places that hold together for minutes. The next frontier is behaviour over years, many lives at once, and results checked against a reality the model has never seen.