• [2025.8.12] Updated RoboTwin2 inference code

H-RDT (Human to Robotics Diffusion Transformer) is a novel approach that leverages large-scale egocentric human manipulation data to enhance robot manipulation capabilities. Our key insight is that large-scale egocentric human manipulation videos with paired 3D hand pose annotations provide rich behavioral priors that capture natural manipulation strategies and can benefit robotic policy learning.

1. Create conda environment:

   conda create -n hrdt python=3.10
   conda activate hrdt

2. Install dependencies:

   pip install -r requirements.txt

3. Download pre-trained models:

   export HF_ENDPOINT=https://hf-mirror.com
   huggingface-cli download --resume-download embodiedfoundation/H-RDT --local-dir ./

Before training, preprocess the EgoDx dataset:

1. Configure paths:

   # Edit datasets/pretrain/setup_pretrain.sh with your paths
   nano datasets/pretrain/setup_pretrain.sh
   
   # Set your EgoDx dataset and T5 model paths:
   export EGODEX_DATA_ROOT="/path/to/your/egodx/dataset"
   export T5_MODEL_PATH="/path/to/your/t5-v1_1-xxl"

2. Setup environment:

   source datasets/pretrain/setup_pretrain.sh

3. Run data processing pipeline:

   # Automatically runs: precompute_48d_actions.py → calc_stat.py → encode_lang_batch.py
   ./datasets/pretrain/run_pretrain_pipeline.sh

After data preprocessing is complete:

1. EgoDx Pretrain (fresh start):

1. Configure dataset:

   # Edit datasets/dataset.py line ~45
   self.dataset_name = "egodx"

2. Run training:

   bash pretrain.sh

2. Pretrain Resume: Edit pretrain.sh, add this line:

--resume_from_checkpoint="checkpoint-450000" \

Pre-computed language embeddings are already provided - no preprocessing needed!

1. Setup environment:

   # Edit datasets/robotwin2/setup_robotwin2.sh if needed (only for regenerating files)
   source datasets/robotwin2/setup_robotwin2.sh

2. Data processing pipeline (Not Required):

   # Not needed - lang_embeddings/ already provided in repository
   # Only run if you want to regenerate files:
   # ./datasets/robotwin2/run_robotwin2_pipeline.sh

1. Configure dataset:

   # Edit datasets/dataset.py line ~45
   self.dataset_name = "robotwin_agilex"  # or your robot name
   
   # Add your dataset initialization if not exists:
   elif self.dataset_name == "your_robot":
       self.hdf5_dataset = YourRobotDataset(config=config)

2. Run training:

   bash finetune.sh  # Already configured with pretrained_backbone_path

Edit your current finetune script, make these changes:

# Change this line:
--mode="finetune" \
# To:
--mode="pretrain" \

# And add:
--resume_from_checkpoint="checkpoint-5000" \

Before training, you need to configure the dataset in datasets/dataset.py:

# In datasets/dataset.py, line ~45
self.dataset_name = "egodx"

# The EgoDxDataset will be automatically initialized

# In datasets/dataset.py, line ~45  
self.dataset_name = "your_robot_name"  # e.g., "robotwin_agilex"

# Add your dataset to the initialization logic:
elif self.dataset_name == "your_robot_name":
    self.hdf5_dataset = YourRobotDataset(
        config=config,
        # your dataset parameters
    )

1. Create your dataset folder: datasets/your_robot/
2. Implement your dataset class (see datasets/robotwin2/ as example)
3. Create data processing scripts (see datasets/pretrain/ or datasets/robotwin2/ as examples)
4. Import in datasets/dataset.py
5. Add initialization logic in VLAConsumerDataset.__init__

• configs/hrdt_pretrain.yaml: Human pre-training configuration
• configs/hrdt_finetune.yaml: Robot fine-tuning configuration
• datasets/dataset.py: Dataset selection and initialization
• Modify state_dim, action_dim, output_size for your robot

Join our WeChat group to discuss H-RDT related technical issues:

WeChat Group QR Code

For other questions or collaboration opportunities, please add personal WeChat:

Personal WeChat QR Code

Note: If the QR code expires, please contact us through project Issues for the latest contact information.