Submission Guidelines

Three commands ship with the challenge release: evaluate runs your policy and writes the logs and metrics, validate runs the same automated checks the organizers run, and package produces the archive you upload. All of them work from one submission directory.

1. Install the challenge release Follow Getting Started. Evaluate from an unmodified checkout of the release tag: the evaluator records a fingerprint of the benchmark source, and a modified tree fails validation.
2. Start from the template Copy the template bundle, then fill in metadata.yaml, put your weights in checkpoint/, and implement policy/policy.py (see Policy Interface).

   cp -r examples/challenge_submission ~/submission

3. Smoke-test on a few episodes Catches interface and loading errors in about a minute. Partial runs are labelled as such and cannot be submitted.

   python -m roboeval.challenge.evaluate --submission ~/submission \
       --tasks LiftPot --episodes 2 --output-dir /tmp/smoke --overwrite

4. Run the full evaluation Runs all 8 tasks on the official seeds and writes logs/ and results.json into the bundle. Each worker evaluates one task; --gpus spreads rendering across GPUs, and --resume continues an interrupted run.

   python -m roboeval.challenge.evaluate --submission ~/submission --workers 16 --gpus 0,1

   For scale, measured with the template policy, which never finishes early (about 155k control steps over the 8 tasks): with its three cameras, the command above took 81 minutes on 2 H100s; proprioception-only took 7 minutes with 8 CPU workers. The longest task sets the pace, since each worker runs one task. Policies that succeed early or use fewer cameras take less; your inference time adds to these figures (a pi0.5 model served from one GPU took 42 minutes).

5. Write the report, then validate and package Add report.pdf. package runs the full validation (including replaying a sample of your logged episodes), writes a checksum manifest, and builds the archive. It refuses to package an invalid bundle.

   python -m roboeval.challenge.validate ~/submission     # optional: package runs it too
   python -m roboeval.challenge.package ~/submission

6. Upload and submit Host the archive somewhere the organizers can download without an account request — Hugging Face Hub (preferred), Google Drive, S3, or an institutional server — and submit the link and the printed SHA-256 through the form.

The bundle has this structure. Files marked generated are written by the tools; do not create or edit them by hand.

submission/
├── metadata.yaml           # you   — team, method, and the interface the evaluator runs
├── report.pdf              # you   — technical report, max 4 pages + refs
├── checkpoint/             # you   — policy weights (or set checkpoint_url, see below)
├── policy/                 # you   — inference code
│   ├── policy.py           #         defines make_policy(), see Policy Interface
│   └── requirements.txt    #         pinned extra dependencies
├── results.json            # generated by evaluate
├── logs/                   # generated by evaluate
│   └── <Family>_<Variation>/
│       ├── episode_<i>.json  # seed, outcome, steps, raw and leaderboard metrics
│       └── episode_<i>.npz   # the exact actions your policy sent
└── MANIFEST.sha256         # generated by package

metadata.yaml

The interface block is not just a declaration: the evaluator builds every environment from it, so what you declare is exactly what runs.

team_name: "Your Team"
method_name: "YourMethod-v2"        # as it should appear on the leaderboard
contact_email: "you@example.edu"
affiliation: "Your Institution"
members:
  - "First Author"
  - "Second Author"
roboeval_release: "corl2026-challenge"

interface:
  action_mode:
    absolute: false                 # false = delta actions
    ee: true                        # true = end-effector pose, false = joint positions
  control_frequency: 20             # Hz; fixed at 20 for the challenge
  observation:
    proprioception: true
    cameras:                        # any of: external, head, front, left_wrist, right_wrist
      - {name: head,        resolution: [256, 256], rgb: true, depth: false}
      - {name: left_wrist,  resolution: [256, 256], rgb: true, depth: false}
      - {name: right_wrist, resolution: [256, 256], rgb: true, depth: false}

policy_seed: 0
test_time_compute: "none"           # e.g. "8 samples, reranked by value head"
training_data:
  roboeval_demos: true              # did you train on the provided demonstrations?
  additional_demos: 0               # count of self-collected demonstrations
  external_pretraining: "OXE, DROID"   # or "none"
training_gpu_hours: 128

open_source: true
code_url: "https://github.com/you/your-method"
checkpoint_url: "https://huggingface.co/you/your-method"

policy/policy.py must define make_policy(checkpoint_dir, metadata), returning an object with reset and act. The evaluator calls make_policy once per worker process. Other modules in policy/ can be imported normally.

def make_policy(checkpoint_dir: Path, metadata: dict) -> Policy: ...

class Policy:
    def reset(self, episode) -> None:
        # episode.family, .variation, .env_name, .instruction (language goal),
        # .seed, .policy_seed, .control_frequency, .max_steps, .action_space
        ...

    def act(self, observation: dict) -> np.ndarray:
        # observation["proprioception"]           joint positions + velocities, both arms
        # observation["proprioception_grippers"]  gripper state
        # observation["rgb_<camera>"]             (3, H, W) uint8, per declared camera
        # observation["depth_<camera>"]           (H, W) float, if depth: true
        # returns an action of shape episode.action_space.shape
        ...

Before each episode the evaluator seeds Python's and (if loaded) PyTorch's RNGs from policy_seed and the episode seed, so a seeded policy reproduces exactly. An exception in act ends that episode as a failure, with the traceback in its log; the run continues.

This file is produced by the evaluator from logs/. Do not edit it by hand: the consistency check re-derives it from your logs, and the replay check re-runs your logged actions in the simulator. It is documented here so you can confirm the evaluator ran over everything it should have.

{
  "roboeval_release": "corl2026-challenge",
  "harness_version": "1.0.0",
  "method_name": "YourMethod-v2",
  "episodes_per_variation": 50,
  "seeds": [1000, 1001, 1002, "..."],
  "interface": { "...": "copied from metadata.yaml" },

  "results": {
    "LiftPot": {
      "PositionAndOrientation": {
        "episodes": 50,
        "metrics": {
          "success": { "mean": 0.62, "std": 0.11 },
          "tp":      { "mean": 0.78, "std": 0.07 },
          "bavd":    { "mean": 0.19, "std": 0.03 },
          "bgvd":    { "mean": 0.08, "std": 0.01 },
          "cpl":     { "mean": 0.94, "std": 0.15 },
          "ct":      { "mean": 4.81, "std": 0.62 },
          "ecc":     { "mean": 1.90, "std": 0.44 },
          "jpl":     { "mean": 5.72, "std": 0.98 },
          "mcj":     { "mean": 5.11, "std": 0.87 },
          "mjj":     { "mean": 27.4, "std": 3.90 },
          "opl":     { "mean": 3.66, "std": 0.71 },
          "scc":     { "mean": 1.12, "std": 0.51 },
          "sc":      { "mean": 0.18, "std": 0.10 },
          "tl":      { "mean": 96.2, "std": 12.4 }
        },
        "terminations": { "success": 31, "fail": 2, "budget": 17, "unhealthy": 0, "error": 0 }
      }
    },

    "LiftTray":                { "...": "..." },
    "StackTwoBlocks":          { "...": "..." },
    "StackSingleBookShelf":    { "PositionAndOrientation": { "...": "..." } },
    "CubeHandover":            { "...": "..." },
    "PackBox":                 { "...": "..." },
    "PickSingleBookFromTable": { "...": "..." },
    "RotateValve":             { "PositionAndOrientation": { "...": "..." } }
  },

  "summary":    { "per_family": { "...": "..." }, "suite": { "...": "..." } },
  "score":      { "overall": { "score": 0.283, "outcome": 0.334, "safety": 0.836, "efficiency": 0.550,
                               "coordination": 0.861 },
                  "tasks":   { "LiftPot": { "...": "..." }, "...": "..." } },
  "provenance": { "git_commit": "...", "source_fingerprint": "...", "mujoco_version": "3.3.3", "...": "..." }
}

Metric keys match the leaderboard schema in static/data/leaderboard_schema.json. Each family has one entry, its PositionAndOrientation variant. All 8 must be present: the schema check rejects an incomplete file, so run your policy on every task even where you expect it to score zero.

A short report is required for every submission — maximum 4 pages plus references, CoRL format. It is not peer-reviewed, and it does not affect your ranking. It exists so the community can learn something from the leaderboard beyond a row of numbers. Cover:

• Method. Architecture, training objective, and what is novel relative to the baselines.
• Data. Which demonstrations you trained on, any data you collected or generated, and any external pretraining.
• Interface. Action mode, observation mode, control frequency, and any test-time compute.
• Diagnostic analysis. Where your policy fails and why, using the stage-level and behavioral metrics. This is the part we most want to read.
• Compute. Training GPU-hours and hardware.

Submissions go through the challenge portal, a Hugging Face Space that also hosts the live leaderboard. Sign in with your Hugging Face account, then paste the link to your archive (or upload it) together with the SHA-256 that package printed, and confirm you have read the rules. Team name, method, and affiliation are read from metadata.yaml.

The portal validates every submission on the spot, with the same checks as validate, and shows you the report within a few minutes. Accepted submissions appear on the leaderboard immediately; the most recent accepted submission is the one ranked. Bundles that fail the automated checks are not counted toward your submission limit, so fix and resubmit.

• Each team submits from one Hugging Face account, and each account belongs to one team. The first accepted submission links them.
• Each team may submit up to 2 times per day (UTC).
• Host the archive somewhere it downloads without signing in, such as a public Hugging Face repo. The portal keeps a copy for verification.

Every submission goes through two automated checks, and prize-eligible submissions go through a third. The automated checks are the ones python -m roboeval.challenge.validate runs, so a bundle that validates locally passes them on receipt.

1. Schema check	Automated. Confirms the bundle layout and metadata.yaml, and that results.json covers all 8 tasks and all 14 metrics on the official seeds, from an unmodified release. Runs on submission; failures are reported back so you can fix and resubmit.
2. Consistency check	Automated. Re-derives every aggregate in results.json from logs/, then replays a random sample of logged episodes' actions in the simulator and requires the same outcome and metrics. This needs no checkpoint. Divergence beyond floating-point tolerance is flagged for manual review.
3. Held-out re-run	Manual, for prize-eligible entries. Organizers run the same evaluator on your bundle with an unpublished seed set (--seeds-file). Results must hold up within normal run-to-run variance.

If step 3 cannot be completed — the checkpoint does not load, dependencies are unresolvable, or the inference code is incomplete — organizers contact you once with a fixed window to supply a working artifact. Entries that remain unreproducible are not eligible for prizes, though they may still be listed on the leaderboard marked as unverified.