Evaluation & Rules

What gets run

Every submission is evaluated on all 8 task families, each on its combined position + orientation variant. There is no task selection: a policy that is excellent on two families and absent on six will rank poorly. Each task–variation pair is run for a fixed number of independent episodes with fixed evaluation seeds.

Task families	8 (all mandatory)
Variations	Position + orientation (PR) of each family; 8 in total, all mandatory
Episodes per variation	50 — identical for every submission
Evaluation seeds	1000–1049 — a fixed list built into the evaluator
Episode termination	On task success, on the task's failure condition, on the step budget, or on an unrecoverable simulator state
Step budget	Simulated seconds per family, independent of control frequency: Lift Pot 10, Lift Tray 10, Pick Book From Table 15, Stack Two Cubes 20, Cube Handover 20, Pack Box 20, Rotate Valve 20, Stack Single Book Shelf 40. About 1.5× the slowest variation's 95th-percentile human demonstration.
Control frequency	Fixed at 20 Hz for every submission, matching the released dataset. Collision and slip counts and jerk depend on the control rate, so a common rate keeps them comparable.
Action modes	Joint positions or end-effector poses, absolute or delta. Every mode is held to the same joint speed limit (2.175 rad/s), so no mode can move the arms faster than another.
Simulator	MuJoCo, bimanual Franka Panda, as pinned in the challenge release tag

Seeds and determinism

Evaluation seeds control initial object poses within each variation's distribution. They are published, not secret — the challenge tests spatial generalization, not seed guessing. Overfitting to the published evaluation seeds is nonetheless a rules violation (see Rules), and organizers re-run leading submissions on a held-out seed set to confirm results hold.

Stage-level scoring

Each task decomposes into ordered, skill-specific stages — for example Stack Single Book Shelf runs push → grasp → lift → place. The evaluator records which stages each episode completed, which produces Task Progression and localizes failures. An episode that grasps correctly but never places is scored differently from one that never made contact, even though both have a success of 0.

Fourteen metrics across four families. ↑ means higher is better, ↓ means lower is better. All are computed by the RoboEval evaluation harness and reported per task, per variation, as mean and standard deviation over episodes.

The leaderboard ranks submissions by the RoboEval Diagnostic Score (RDS), a number between 0 and 1 that combines outcome with how well the policy behaved. It is computed per task from the task's mean metrics and then averaged over the 8 tasks with equal weight. The evaluator prints it, and results.json contains it, so you can see your score before submitting.

Weights

Outcome	40% — outcome score = 0.7 × Success + 0.3 × Task Progression
Safety & Stability	30% — SCC, ECC, SC, MJJ, MCJ
Efficiency	20% — CPL, JPL, OPL, CT (TL is CT × 20 at the fixed rate and is not counted again)
Bimanual Coordination	10% — BAVD, BGVD

Normalizing behavior against humans

Raw metric values are not comparable across tasks — Stack Single Book Shelf trajectories are three times longer than Lift Pot trajectories. Each behavioral metric is therefore scored against a human reference: the median of that metric over the human demonstrations of the same task in the training dataset (roboeval_ee_delta_20hz_scaled) that succeed when replayed under the challenge rules (20 Hz, same step budget, stopping at success). A value at or better than the reference scores 1. A worse value v against reference r scores (r + c) / (v + c), about 0.5 at twice the reference, where c is 10% of the reference (at least one event for collision and slip counts). A family's score is the mean of its metrics' scores. The references ship with the release in roboeval/challenge/human_reference.json.

Combining outcome and behavior

For each task, with outcome score O and family scores S (safety), E (efficiency) and C (coordination):

task score = 0.4 × O + O × (0.3 × S + 0.2 × E + 0.1 × C)

Behavior counts in proportion to outcome. A policy that does nothing has spotless path lengths and no collisions but scores 0; one that always succeeds with human-level behavior scores 1.

Tie-breaking

Ties on RDS are broken in order by: (1) mean Success, (2) mean Task Progression, (3) Safety & Stability family score, (4) earlier submission timestamp.

Secondary leaderboards

Alongside the overall ranking, the portal shows the RDS for each task and views by metric family, and can filter entries by observation mode (camera-based or proprioception-only) and action mode. Prizes follow the overall ranking only; per-task boards are the intended reference for papers reporting on a subset of RoboEval.

Who can enter

• Open to anyone: academic, industrial, and independent participants are all welcome.
• Teams may be of any size. Each team submits under a single identity.
• An individual may belong to only one team.
• Challenge organizers and their immediate research groups may submit entries for reference, but are not eligible for prizes and are marked as such on the leaderboard.

What is allowed

• Any policy architecture, including vision-language-action models, diffusion policies, and classical or hybrid controllers.
• Any pretraining, including on external robotics datasets and internet-scale corpora.
• Collecting additional demonstrations yourself with the provided teleoperation tools.
• Data augmentation, synthetic data generation, and domain randomization during training.
• Using simulator state offline to generate, label, or filter training data — for example scripted experts that read object poses, or training rewards computed from success and stage signals. Only the evaluated policy is restricted to its declared observations; describe such data in your report.
• Task-conditioned or per-task models, provided the same submission covers all 8 tasks.
• Test-time compute such as sampling and reranking, declared in metadata.yaml. Evaluation is not real-time, so inference speed does not affect scores.

What is not allowed

• Reading privileged simulator state during evaluation that is not exposed through the standard observation interface — ground-truth object poses in an image-only submission, contact flags, or internal solver state.
• Modifying the environment, task definition, reward, or termination conditions. The challenge release tag is the reference; local edits to it invalidate a submission. The evaluator records a fingerprint of the benchmark source, and validation rejects a modified tree. Rewards you compute in your own training code are up to you.
• Tuning on the published evaluation seeds. Use them to sanity-check, not to select checkpoints or hyperparameters.
• Hard-coding trajectories keyed to specific initial states rather than learning or computing a policy from observations.
• Editing the metrics computation or post-processing results.json by hand.
• Multiple accounts used to circumvent the submission limit.

Submission limits

Each team may submit up to 2 entries per day (UTC); submissions rejected by the automated checks do not count. The most recent valid submission is the one ranked; earlier entries remain visible in the team's history. There is no limit on running the evaluation harness locally.

Every submission must be reproducible by the organizers from the artifacts provided. Concretely:

• The policy checkpoint must load and run against the pinned challenge release with the declared dependencies.
• The technical report must state the training data used, the architecture, the action and observation modes, the control frequency, and any test-time compute.
• Randomness in the policy must be seedable, and the seed must be declared as policy_seed in metadata.yaml.
• Total training compute should be reported in GPU-hours. This does not affect ranking; it is collected so the community can compare methods fairly.

Full artifact requirements are specified on the Submission Format page.

RoboEval — the benchmark, assets, and demonstration data — is released under the MIT License. You retain full ownership of your policy, code, and checkpoints. Submitting grants the organizers permission to run your submission for evaluation and to publish the resulting metrics, your team name, and your method description on the leaderboard and in any workshop report summarizing the challenge.

Releasing your code and checkpoints is encouraged and qualifies for the open-source award, but is not required to compete.

All participants are expected to follow the CoRL 2026 code of conduct. Report concerns to the track organizers at yiruwang@cs.washington.edu.