RoboDojo
How it works
The benchmark provides a simulator client and a suite of evaluation tasks running on Isaac Sim, with heterogeneous parallel simulation of different tasks and scenes for fast, scalable feedback. Scenes are built from physically grounded, configuration-driven assets (rigid, articulated, and deformable objects), with seed-controlled layouts for reproducibility. The RoboDojo-RealEval layer runs analogous tests on physical robots (Piper X, Piper, ARX X5) with remote cloud access, standardized hardware, and scene reset. Policies are attached through the shared XPolicyLab interface, and a single command (summarize) aggregates results into a leaderboard table. In the current release RoboDojo is eval-only — it ships the simulator, tasks, asset/config validation, and result artifacts, while policy integration and policy servers belong to XPolicyLab.
Problem solved
Prior manipulation benchmarks were typically sim-only or real-only, which made scalable, reproducible, and comparable evaluation of generalist policies difficult. RoboDojo bridges both regimes and standardizes the evaluation protocol so results are reproducible and comparable across models.
Components
42 Isaac Sim tasks grouped into five capability dimensions (generalization, memory, precision, long-horizon, open tasks), with heterogeneous parallel simulation.
18 tasks on physical robots (Piper X, Piper, ARX X5) with remote cloud access, standardized hardware, scene reset, an evaluation protocol, and a deployment interface.
Companion project unifying many policies behind a single interface, used for both simulation and real-world runs; in the eval-only RoboDojo release, policy integration belongs to XPolicyLab.
Public, continuously updated ranking; results aggregated with a single summarize command over seed-controlled layouts for reproducibility.
Implementation
RoboDojo ships only the simulator, tasks, asset/config validation, and result artifacts; policy integration and policy servers are not part of the repository.
Simulation relies on Isaac Sim and requires sufficient GPU memory; the documentation lists common issues with installation, assets, and GPU memory.
Evolution
RoboDojo was published on arXiv (2607.04434) and its repository was open-sourced; the benchmark ships as eval-only.