Dualsense™ Controller Teleoperation and ACT autonomy IL on ALOHA for one Bigym benchmark task

Outcomes:

1. The ACT model successfully completed the task.
2. Implemented a multi-buffer inference system for improved movement stability, and applied an inference result scaler to address action attenuation—ensuring that the movement amplitude was sufficient to avoid object penetration in this scenario 100% avoided.
3. Integrated MuJoCo Warp in headless mode, achieving a 10x increase in step speed.

Intro:

• Insipred by the Nitendo-Aloha
• Reproduce the implementation and concept of Nintendo-Aloha and replace the controller to be compatible with Dualsense™ Controller
• Explore ACT IL, Mujoco, Warp
• Original Readme of bigym

Teleoperation, refer to the ds_aloha.py script,based on pydualsense library.

Model training repository

Model inference entry point: controllers/demonstration/run_inference_temporalAgg.py

After ACT model training, action outputs are often weaker than intended, leading to less accurate positioning (though movement direction is usually correct). This is likely due to most collected actions being near zero. To compensate, we apply a scaling factor to the inferred actions so the robot arm moves sufficiently. Comparing action distributions between the dataset and policy outputs is recommended.

Binary controller actions (e.g., action2, action9) can become diluted during training. To address this, we use MixUp interpolation to convert discrete actions into smoother, continuous values, improving model learning and inference.

MuJoCo Warp is optimized for fast, headless multi-world simulations and is not ideal for single-world, visualized policy inference.

Model training throug colab by A100 about 2s per it. vs my RTX3060 17s per it.