Robots practise among lives

A robot that learns to move among people learns from whatever people it practises with. In most robot training those people are scripted: they walk straight lines, or push away from each other by a simple rule. A policy trained among them learns to handle them, and then meets real people, who do neither.

Living Worlds already answered the first half of that problem. Its crowd predicted how close real people came to a real robot in a room where visitors carried boxes, buckets and large objects, on recording days nobody here had opened, better than a crowd blind to the robot in 7 of 8 recordings. That test was registered before any of those days was read.

The same crowd, in Isaac Lab

That crowd now runs inside NVIDIA Isaac Lab 3.0 as a task of its own, LW-CrowdNav-Direct. A robot takes the place of one person in a scene and has to reach where that person went, through the same people. Everything else is the robots test as it was run: the same episodes, the same robot (a disc of 0.3 m, at most 1.2 m/s and 1.0 m/s²), the same people, and the same rule for contact, the closest approach to each person within every step, computed exactly.

The difference is scale. Isaac Lab steps thousands of these episodes at once on the GPU, with the Newton physics engine and without a full Isaac Sim installation, and trains a policy with standard reinforcement learning libraries.

Checked before it is used

A faster copy of a test is only worth having if it is the same test. So the two were run side by side:

• The task finds the same 633 episodes in the eight simulated scenes as the original.
• With the same scripted policy on 200 episodes, both give the same result for every episode: whether the robot arrived, which people it touched and which it nearly touched.

Only then was anything trained. On our own GPU, 1,024 environments trained for 150 iterations, about 4.9 million robot steps, in about a minute, and the trained policy plays back among the crowd as video. These are development runs. The registered test of a trained robot among real people, on days held out, comes next, and its result will be published whatever it shows.

The safety layer, checked the same way

A robot among people also carries a safety layer that watches it from outside: cameras on the walls, a perception model, and a decision maker that tells the robot to slow down or stop. NVIDIA Halos is a reference design for exactly that. Its Safety Core takes per-frame positions of people and machines and decides, pair by pair, whether a person is close enough for the robot to slow (within 2 m) or stop (within 1 m).

We ran Halos' Safety Core on our own GPU and gave it Living Worlds scenes as its eyes: every 0.4 s, the robot and the person nearest to it, with their exact positions. Because the true distance is known at every moment, every command it sent can be scored.

• Every command matched the true distance as it was 0.8 s earlier, and none was less cautious than that.
• Scored against the distance at the moment the command left, the match falls, because the scene has already moved on. At 1.2 m/s, 0.8 s is about a metre: on approach, the 2 m and 1 m zones act as if they began a metre later.
• With several people in view, the decision maker acts on whichever person it reads last, not the closest. Sent the nearest person only, it decides on the pair that matters.

None of this is visible without knowing where everyone really was. That is what a simulated crowd gives a safety layer before it ever watches a real one.

Why the crowd comes first

Simulators for robots are no longer the bottleneck. Isaac Lab, Newton and the engines beside them make physics, contact and rendering fast and good. What a simulator cannot supply on its own is people who behave like people around a machine. That is what Living Worlds measures, and it is why the crowd was checked against real recordings before a single robot was trained among it.

The same tasks run on the Lisgen fleet for those in its early access: train among Living Worlds crowds, or among your own recorded scenes, and get the run, its checkpoints and its video back.