Yi Ru Wang1 · Carter Ung1 · Jiafei Duan1,2 · Markus Grotz1 · Wilbert Pumacay1 · Ranjay Krishna1,2 · Dieter Fox1 · Siddhartha Srinivasa1
1University of Washington · 2Allen Institute for AI
Binary success rates hide how a bimanual policy fails. This challenge asks you to build policies that are not just successful but well-behaved — coordinated, efficient, safe, and robust to spatial variation — and scores them with stage-level diagnostics rather than a single pass/fail bit.
Part of the CoRL 2026 Embodied AI Benchmark and Sim-to-Real Challenge Suite, hosted at the workshop From GPU-Accelerated Simulation to Scalable and Generalizable Real Robot Policy Learning — November 12, 2026, Austin, Texas, USA.
| Track | Track 2 of 3 — Diagnostic Bimanual Manipulation |
|---|---|
| Simulator | MuJoCo, bimanual Franka Panda |
| Tasks | 8 task families, each evaluated on its combined position + orientation variant; demonstrations cover every variant |
| Demonstrations | 3,000+ human demonstrations collected via VR and keyboard teleoperation |
| Scored on | RoboEval Diagnostic Score: outcome (40%), safety & stability (30%), efficiency (20%), and bimanual coordination (10%), from 14 metrics compared against human demonstrations |
| What you submit | Policy checkpoint, evaluation logs, and a short technical report |
| Baselines | ACT, Diffusion Policy, GR00T (see the leaderboard) |
| License | MIT — benchmark, assets, and demonstrations are free to use |
Deadline
Submissions close at 23:59 Anywhere on Earth (UTC−12) on November 1, 2026, which is 11:59 UTC on November 2. The portal stops accepting submissions automatically at that time. Watch the RoboEval repository or the workshop site for announcements.
Awards follow the overall leaderboard ranking; winners present their approach at the workshop on November 12, 2026.
Open-source bonus
Entries that release training code and checkpoints under a permissive license are eligible for an additional open-source award and are highlighted on the public leaderboard. We strongly encourage reproducible submissions — the point of a diagnostic benchmark is that others can build on what you learned.
In our study, policies with nearly identical success rates diverged sharply in how they executed the same task — some lost alignment, others lost temporally consistent bimanual control. Behavioral metrics correlated with success in over half of all task–metric pairs, and stayed informative even where binary success saturated. This challenge is built around that finding.
Every task decomposes into skill-specific stages, so a failure is localized to where it happened — not just recorded as a zero.
Each task is evaluated on its combined position + orientation variant, so object poses change in every episode and a policy must generalize spatially rather than memorize a fixed scene.
Arm velocity mismatch and gripper height offset are first-class metrics, so genuinely bimanual behavior is rewarded over two arms acting independently.
Collisions, slips, and jerk are measured throughout. A policy that succeeds by flailing through the scene will not win.
The challenge uses the RoboEval task suite: everyday bimanual activities drawn from service, warehouse, and industrial settings, each paired with expert demonstrations and stage annotations. The challenge evaluates every task family on its PR (combined position and rotation) variant; demonstrations from all variants listed below may be used for training.
| Task Family | Variations | # Demos | Traj. Len | Skills | Coordination Type |
|---|---|---|---|---|---|
| Lift Tray | Static, Pos, Rot, PR | 400 | 117.3 | grasp, lift | Tight Sym. |
| Stack Two Cubes | Static, Pos, Rot, PR | 492 | 120.9 | grasp, hold, place | Loosely Coord. |
| Stack Single Book Shelf | Static, Pos, PR | 210 | 216.8 | push, grasp, lift, place | Loosely Coord. |
| Cube Handover | Static, Pos, Rot, PR | 406 | 101.8 | grasp, hold | Loosely Coord. |
| Lift Pot | Static, Pos, Rot, PR | 421 | 93.1 | grasp, lift | Tight Sym. |
| Pack Box | Static, Pos, Rot, PR | 423 | 183.8 | push | Uncoord. |
| Pick Book From Table | Static, Pos, Rot, PR | 371 | 131.4 | grasp, lift | Loosely Coord. |
| Rotate Valve | Static, Pos, PR | 337 | 154.9 | grasp, rotate along axis | Uncoord. |
Pos = position-only variation · Rot = orientation-only variation
· PR = combined position and rotation. # Demos and Traj. Len (mean episode length in control steps at 20 Hz) are
for the training dataset, roboeval_ee_delta_20hz_scaled, over all
variants of each task: 3,060 episodes in total.
Over 3,000 human demonstrations, collected through VR teleoperation on an Oculus Quest and through
keyboard teleoperation, are released as a ready-to-train LeRobot dataset,
roboeval_ee_delta_20hz_scaled: 3,060 episodes
over all 30 variants at 20 Hz, with end-effector delta actions, five 256×256 cameras,
proprioception, and a language instruction per episode. See
Getting Started to download and load it. You may use all of it, a subset,
or none of it.
Actions are joint positions or end-effector poses, absolute or delta, for both arms plus grippers, applied at 20 Hz, with the same joint speed limit in every mode. Observations are proprioception (joint positions and velocities, gripper state) plus RGB and/or depth from any of five cameras: external, head, front, and one on each wrist. You declare the combination in your submission and the evaluator runs exactly that.
Submissions are ranked by the RoboEval Diagnostic Score, which combines four metric families: outcome 40%, safety & stability 30%, efficiency 20%, and bimanual coordination 10%. Behavior is scored against human demonstrations of the same task and counts in proportion to how often the policy succeeds, so a policy that does nothing scores 0. The evaluator prints your score. The formula is in Ranking and the metric definitions are in the Metrics Reference.
git clone https://github.com/Robo-Eval/RoboEval.git
cd RoboEval
git checkout corl2026-challenge
git submodule update --init
conda create -n roboeval python=3.10 -y
conda activate roboeval
pip install -e ".[examples]"export MUJOCO_GL=egl # on machines without a display
python -m roboeval.challenge.evaluate --submission examples/challenge_submission \
--tasks LiftPot --episodes 2 --output-dir /tmp/roboeval_check --overwrite
With a display attached, python examples/1_data_replay.py also
opens the viewer and replays human demonstrations. Its first run downloads about 1.8 GB of demos.
import numpy as np
from roboeval.envs.lift_pot import LiftPotPositionAndOrientation
from roboeval.action_modes import JointPositionActionMode
from roboeval.robots.configs.panda import BimanualPanda
from roboeval.utils.observation_config import CameraConfig, ObservationConfig
env = LiftPotPositionAndOrientation(
# end-effector delta actions, as in the released dataset
action_mode=JointPositionActionMode(absolute=False, ee=True,
floating_base=True, floating_dofs=[]),
observation_config=ObservationConfig(
cameras=[CameraConfig(name="head", resolution=(256, 256))]),
render_mode=None, # "human" opens a viewer and needs a display
control_frequency=20,
robot_cls=BimanualPanda,
)
obs, info = env.reset(seed=1000)
obs, reward, terminated, truncated, info = env.step(np.zeros(env.action_space.shape))
print(obs.keys(), info["subtask_progress"])lerobot needs NumPy 2 while RoboEval pins
NumPy 1.26, so train in a separate environment from the one you evaluate in.
conda create -n roboeval-train python=3.11 -y
conda activate roboeval-train
pip install lerobot huggingface_hub
# the 8 evaluated variants (~31 GB); drop --include for all 30 (~102 GB)
hf download helen9975/roboeval_ee_delta_20hz_scaled --repo-type dataset \
--include "*PositionAndOrientation/*" --local-dir roboeval_datafrom lerobot.datasets.lerobot_dataset import LeRobotDataset
ds = LeRobotDataset("LiftPotPositionAndOrientation",
root="roboeval_data/LiftPotPositionAndOrientation", revision="local")
frame = ds[0] # cam_head, cam_front, cam_external, cam_left_wrist, cam_right_wrist,
# state (36), state_gripper (2), actions (14), task (instruction)
The dataset matches the template submission's interface (end-effector delta actions at 20 Hz). At
evaluation time your policy receives rgb_<camera> images
(3×H×W, uint8) for the cameras you declare,
proprioception (the dataset's state),
proprioception_grippers (state_gripper),
and the task's instruction. Train with any framework (for example LeRobot's policies or openpi), on
these demonstrations, your own data, or any pretraining; see the
rules for what is and is not permitted.
policy/policy.py at your model,
declare your interface in metadata.yaml, and run all 8 tasks on
the official seeds.
cp -r examples/challenge_submission ~/submission
python -m roboeval.challenge.evaluate --submission ~/submission --workers 16 --gpus 0python -m roboeval.challenge.package ~/submissionDocumentation
Hardware note
RoboEval runs on Linux with MuJoCo. Proprioception-only policies evaluate on CPU; camera observations are rendered on an NVIDIA GPU through EGL. VR teleoperation for collecting your own demonstrations additionally requires an Oculus Quest and GLIBC 2.32+.
Questions?
Open an issue on the RoboEval repository or email the track organizers — see Contact below.
This track is organized by the RoboEval team at the University of Washington and the Allen Institute for AI.
Yi Ru Wang1 · Carter Ung1 · Jiafei Duan1,2 · Markus Grotz1 · Wilbert Pumacay1 · Ranjay Krishna1,2 · Dieter Fox1 · Siddhartha Srinivasa1
1University of Washington · 2Allen Institute for AI
Track questions: yiruwang@cs.washington.edu
Workshop questions: zhenzhenl@nvidia.com
Technical issues: github.com/Robo-Eval/RoboEval/issues
The challenge suite runs three independent tracks under a shared reporting standard. Entering one does not require entering the others.
1,000 household activities across 50 scenes and 10,000 objects in Isaac Sim, with long-horizon navigation and bimanual manipulation.
Stage-level failure diagnosis for bimanual manipulation in MuJoCo, scored on behavior as well as outcome.
Policies trained in GPU-accelerated simulation must generalize to physical robots across multiple real-world embodiments.
If you use RoboEval or participate in this challenge, please cite:
@misc{wang2025roboevalroboticmanipulationmeets,
title={RoboEval: Where Robotic Manipulation Meets Structured and Scalable Evaluation},
author={Yi Ru Wang and Carter Ung and Grant Tannert and Jiafei Duan and Josephine Li
and Amy Le and Rishabh Oswal and Markus Grotz and Wilbert Pumacay
and Yuquan Deng and Ranjay Krishna and Dieter Fox and Siddhartha Srinivasa},
year={2025},
eprint={2507.00435},
archivePrefix={arXiv},
primaryClass={cs.RO}
}