The embodied artificial intelligence startup X Square Robot has released X2Real—a simulation benchmark designed to measure the performance of fundamental level robot models in the real world, as well as to continuously expand the test dataset as models improve.
The company clearly states the problem: evaluating real robots is costly and slow, whereas simulation is fast but often unrealistic and allows models to easily overfit. X2Real is based on three principles: fidelity, efficiency, and fairness.
To ensure fidelity, the team recorded trajectories from physical robotic manipulators, reproduced the same control commands in the simulation, and tuned joint responses until they matched, which the company claims ensures trajectory consistency at the millimeter level. Visual appearance and physical characteristics, such as contact, were also calibrated. Supported embodiments include the single-arm Franka, the two-arm ArtiXon Arm-6A, ARX R5, and the mobile manipulator Quanta X1. When performing general manipulation tasks, X Square Robot reports a correlation of 0.96 between task success in simulation and in the real world, and 0.99 for progress metrics. This data was obtained through the company's own tests on this subset of tasks.
Regarding efficiency, X2Real includes 44 tasks distributed across 10 categories. Four categories are dedicated to reasoning: generalization, visual understanding, language comprehension, and memory. Five categories relate to manipulation: fine manipulation, two-arm coordination, tool use, dynamic objects, and mobile manipulation. A separate set of tasks, including block stacking, pressing buttons in a specified color sequence, and solving the Tower of Hanoi, is aimed at testing accuracy, long-term memory, and planning. The company notes that most existing models still struggle with these tasks.
The principle of fairness is achieved by independently randomizing the appearance, task setup, and embodiment conditions. Models trained after this stage use a dataset collected over nearly 300 hours in a studio with a uniform green screen, which is separate from the evaluation scenes. The benchmark operates within the Mana simulation ecosystem, built on NVIDIA Isaac Lab-Arena, which packages scenes, robots, and task steps into reusable modules.
AI agents manage two loops. One generates new tasks and scenes from a limited menu of options with automatic checking to exclude incorrect settings. The second breaks down each task into a subgraph of subtasks, evaluates each node based on object poses, contacts, and grasp states, and creates intentionally new tests where the model fails.
A unified protocol called Policy Space standardizes differences in input formats, action representations, and output timing. Developers perform inference locally and connect to a platform that handles online evaluation and trajectory replay without requiring a local simulation setup.