• --task: Specify the task environment name to run, e.g., LeIsaac-SO101-PickOrange-v0.

• --seed: Specify the seed for environment, e.g., 42.

• --teleop_device: Specify the teleoperation device type, e.g., so101leader, bi-so101leader, keyboard.

• --port: Specify the port of teleoperation device, e.g., /dev/ttyACM0. Only used when teleop_device is so101leader.

• --left_arm_port: Specify the port of left arm, e.g., /dev/ttyACM0. Only used when teleop_device is bi-so101leader.

• --right_arm_port: Specify the port of right arm, e.g., /dev/ttyACM1. Only used when teleop_device is bi-so101leader.

• --num_envs: Set the number of parallel simulation environments, usually 1 for teleoperation.

• --device: Specify the computation device, such as cpu or cuda (GPU).

• --enable_cameras: Enable camera sensors to collect visual data during teleoperation.

• --record: Enable data recording; saves teleoperation data to an HDF5 file.

• --dataset_file: Path to save the recorded dataset, e.g., ./datasets/record_data.hdf5.

• --resume: Enable resume data recording from the existing dataset file.

• --task_type: Specify task type. If your dataset is recorded with keyboard, you should set it to keyboard, otherwise not to set it and keep default value None.

• --quality: Whether to enable quality render mode.

• --task: Specify the task environment name to run, e.g., LeIsaac-SO101-PickOrange-v0.

• --num_envs: Set the number of parallel simulation environments, usually 1 for replay.

• --device: Specify the computation device, such as cpu or cuda (GPU).

• --enable_cameras: Enable camera sensors to visualize when replay.

• --replay_mode: Replay mode, we support replay action or state.

• --task_type: Specify task type. If your dataset is recorded with keyboard, you should set it to keyboard, otherwise not to set it and keep default value None.

• --dataset_file: Path to the recorded dataset, e.g., ./datasets/record_data.hdf5.

• --select_episodes: A list of episode indices to replayed, Keep empty to replay all episodes.

• --task: Name of the task environment to run for inference (e.g., LeIsaac-SO101-PickOrange-v0).

• --seed: Seed of environment (default: current time).

• --episode_length_s: Episode length in seconds (default: 60).

• --eval_rounts: Number of evaluation rounds. 0 means don't add time out termination, policy will run until success or manual reset (default: 0)

• --policy_type: Type of policy to use (default: gr00tn1.5).

  • now we support gr00tn1.5, lerobot-<model_type>

• --policy_host: Host address of the policy server (default: localhost).

• --policy_port: Port of the policy server (default: 5555).

• --policy_timeout_ms: Timeout for the policy server in milliseconds (default: 5000).

• --policy_action_horizon: Number of actions to predict per inference (default: 16).

• --policy_language_instruction: Language instruction for the policy (e.g., task description in natural language).

• --policy_checkpoint_path: Path to the policy checkpoint (if required).

• --device: Computation device, such as cpu or cuda.

You may also use additional arguments supported by IsaacLab's AppLauncher (see their documentation for details).

For usage instructions, please refer to the examples provided above.

We utilize lerobot's async inference capabilities for policy execution. For detailed information, please refer to the official documentation. Prior to execution, ensure that the policy server is running.

python scripts/evaluation/policy_inference.py \
    --task=LeIsaac-SO101-PickOrange-v0 \
    --policy_type=lerobot-smolvla \
    --policy_host=localhost \
    --policy_port=8080 \
    --policy_timeout_ms=5000 \
    --policy_language_instruction='Pick the orange to the plate' \
    --policy_checkpoint_path=outputs/smolvla/leisaac-pick-orange/checkpoints/last/pretrained_model \
    --policy_action_horizon=50 \
    --device=cuda \
    --enable_cameras

We utilize openpi's remote inference capabilities for policy execution. For detailed information, please refer to the official documentation. Prior to execution, ensure that the policy server is running.

python scripts/evaluation/policy_inference.py \
    --task=LeIsaac-SO101-PickOrange-v0 \
    --policy_type=openpi \
    --policy_host=localhost \
    --policy_port=8000 \
    --policy_timeout_ms=5000 \
    --policy_language_instruction='Pick the orange to the plate' \
    --device=cuda \
    --enable_cameras

Inspired by the greenscreening functionality from SIMPLER and ManiSkill, we have implemented DigitalTwin Env. This feature allows you to replace the background in simulation environments with real background images while preserving the foreground elements such as robotic arms and interactive objects. This approach significantly reduces the gap between simulation and reality, enabling better sim2real transfer.

To use this feature, simply create a task configuration class that inherits from ManagerBasedRLDigitalTwinEnvCfg and launch it through the corresponding environment. In the configuration class, you can specify relevant parameters including overlay_mode, background images path, and foreground environment components to preserve.

For usage examples, please refer to the sample task: LiftCubeDigitalTwinEnvCfg.

We have integrated IsaacLab MimicGen, a powerful feature that automatically generates additional demonstrations from expert demonstrations.

To use this functionality, you first need to record some demonstrations. Recording scripts can be referenced from the instructions above. (Below we use the MimicGen for the LeIsaac-SO101-LiftCube-v0 task as an example).

[NOTE] Pay attention to the input_file and output_file parameters in the following scripts. Typically, the output_file from the previous script becomes the input_file for the next script.

Since MimicGen requires trajectory generalization based on end-effector pose and object pose, we first convert joint-position-based action data to IK-based action data. The conversion process is as follows, where input_file specifies the collected demonstration data:

python scripts/mimic/eef_action_process.py \
    --input_file ./datasets/mimic-lift-cube-example.hdf5 \
    --output_file ./datasets/processed_mimic-lift-cube-example.hdf5 \
    --to_ik --headless

Next, we perform sub-task annotation based on the converted action data. Annotation can be done in two ways: automatic and manual. If you want to use automatic annotation, please add the --auto startup option.

python scripts/mimic/annotate_demos.py \
    --device cuda \
    --task LeIsaac-SO101-LiftCube-Mimic-v0 \
    --input_file ./datasets/processed_mimic-lift-cube-example.hdf5 \
    --output_file ./datasets/annotated_mimic-lift-cube-example.hdf5 \
    --enable_cameras

After annotation is complete, we can proceed with data generation. The generation process is as follows:

python scripts/mimic/generate_dataset.py \
    --device cuda \
    --num_envs 1 \
    --generation_num_trials 10 \
    --input_file ./datasets/annotated_mimic-lift-cube-example.hdf5 \
    --output_file ./datasets/generated_mimic-lift-cube-example.hdf5 \
    --enable_cameras

After obtaining the generated data, we also provide a conversion script to transform it from IK-based action data back to joint-position-based action data, as follows:

python scripts/mimic/eef_action_process.py \
    --input_file ./datasets/generated_mimic-lift-cube-example.hdf5 \
    --output_file ./datasets/final_generated_mimic-lift-cube-example.hdf5 \
    --to_joint --headless

Finally, you can use replay to view the effects of the generated data. It's worth noting that due to the inherent randomness in IsaacLab simulation, the replay performance may vary.

[NOTE] Depending on the device used to collect the data, you need to specify the corresponding task type with --task_type. For example, if your demonstrations were collected using the keyboard, add --task_type=keyboard when running annotate_demos and generate_dataset.

task_type dose not need to be provided during replay the results final_generated_dataset.hdf5.