Welcome to RoboEval’s Documentation#
RoboEval is a structured benchmark for bimanual robotic manipulation that provides:
8 task families with 28 total variations
3,000+ human-collected demonstrations via VR and keyboard teleoperation
Rich diagnostic metrics beyond binary success (coordination, efficiency, safety, task progression)
Standardized tools for data collection, conversion, and evaluation
Read our paper: RoboEval: Where Robotic Manipulation Meets Structured and Scalable Evaluation
Key Features#
- Diverse Bimanual Tasks
From basic manipulation (StackTwoBlocks) to complex coordination (LiftPot, PackBox)
- Comprehensive Metrics
Track task progression, coordination quality, trajectory efficiency, and spatial proximity
- Flexible Data Collection
VR teleoperation (Oculus Quest) and keyboard control for high-quality demonstrations
- Easy Integration
Built on Gymnasium API with support for multiple action modes and observation configs
- Extensible
Add custom tasks, props, robots, and metrics with well-documented APIs
Quick Start#
from roboeval.envs.lift_pot import LiftPot
from roboeval.action_modes import JointPositionActionMode
from roboeval.robots.configs.panda import BimanualPanda
env = LiftPot(
action_mode=JointPositionActionMode(),
render_mode="human",
robot_cls=BimanualPanda
)
obs, info = env.reset()
for _ in range(1000):
action = env.action_space.sample()
obs, reward, terminated, truncated, info = env.step(action)
if terminated or truncated:
break
Getting Started
- Installation
- Quick Start
- Examples
- Example 1: Data Replay
- Example 2: Convert and Replay
- Example 3: Load, Convert, and Replay
- Example 4: Evaluate OpenVLA
- Example 5: Gather Metrics
- Example 6: Data Collection (Keyboard)
- Example 7: Data Collection (Oculus VR)
- Quick Start Guide
- Running Examples in Sequence
- Additional Usage Patterns
- Tips and Best Practices
- Next Steps
Tasks & Environments
Advanced Topics
API Reference
Development