There are 4 steps to run this code
- Python Virtual Environment and dependencies
- Data download and preprocessing
- Training
- Test and Animation
Anaconda is recommended to create the virtual environment
conda env create -f environment.yaml
source activate torch-action2pose
pip install bert4kerasWe use three datasets and they are: HumanAct12, NTU-RGBD and CMU Mocap. All datasets have been properly pre-transformed to better fit our purpose. Details are provided in our project webpage or dataset documents.
If you just want to play our pre-trained models without Lie version, you don't need to download datasets.
Create a folder for dataset
mkdir ./dataset/If you'd like to use HumanAct12 dataset, download the data folder here, and place it in dataset/
If you'd like to use NTU-RGBD dataset, download the data folder here, and place it in dataset/
If you'd like to use CMU-Mocap dataset, download the data folder here, and place it in dataset/
Our pre-trained models have been involved in folder checkpoints/. You don't need to download them additionally.
If you just want to play our pre-trained models, you could skip this step.
We train the models using the script train_motion_vae.py. All the argments and their descriptions used for training are given in options/base_vae_option.py and options/train_vae_option.py. Some of them were used during trials, but may not be used in our paper. The argments used in examples are these which produce best performances during tuning.
- HumanAct12
python train_motion_vae.py --name <Experiment_name> --dataset_type humanact12 --batch_size 128 --motion_length 60 --coarse_grained --lambda_kld 0.001 --eval_every 2000 --plot_every 50 --print_every 20 --save_every 2000 --save_latest 50 --time_counter --use_lie --gpu_id 0 --iters 50000All motions are of length 60.
- NTU-RGBD
python train_motion_vae.py --name <Experiment_name> --dataset_type ntu_rgbd_vibe --batch_size 128 --motion_length 60 --lambda_kld 0.01 --eval_every 2000 --plot_every 50 --print_every 20 --save_every 2000 --save_latest 50 --time_counter --use_lie --gpu_id 0 --iters 50000 All motions are of length 60.
- CMU Mocap
python train_motion_vae.py --name <Experiment_name> --dataset_type mocap --batch_size 128 --motion_length 100 --lambda_kld 0.01 --eval_every 2000 --plot_every 50 --print_every 20 --save_every 2000 --save_latest 50 --time_counter --use_lie --gpu_id 0 --iters 50000 All motions are of length 100.
Model files and intermediate data will be stored in ./checkpoints
If you are generating results from models with Lie representation, you need to download the corresponding datasets and place them in/dataset. Because our model need to sample skeletons from real human datasets.
The animation results will appear in eval_results/
- HumanAct12
python evaluate_motion_vae.py --name vanilla_vae_lie_mse_kld001 --dataset_type humanact12 --use_lie --time_counter --motion_length 60 --coarse_grained --gpu_id 0 --replic_times 5 --name_ext _R0- NTU-RGBD
python evaluate_motion_vae.py --name vanilla_vae_lie_mse_kld01 --dataset_type ntu_rgbd_vibe --use_lie --time_counter --motion_length 60 --gpu_id 0 --replic_times 5 --name_ext R0 - CMU Mocap
python evaluate_motion_vae.py --name vanilla_vae_lie_mse_kld01 --dataset_type mocap --use_lie --time_counter --motion_length 60 --gpu_id 0 --replic_times 5 --name_ext R0 - HumanAct12
python evaluate_motion_vae.py --name vanila_vae_tf --dataset_type humanact12 --motion_length 60 --coarse_grained --gpu_id 0 --replic_times 5 --name_ext R0
- NTU-RGBD
python evaluate_motion_vae.py --name vanila_vae_tf_2 --dataset_type ntu-rgbd-vibe --motion_length 60 --gpu_id 0 --replic_times 2 --name_ext R0 - CMU Mocap
python evaluate_motion_vae.py --name vanila_vae_tf_2 --dataset_type mocap --motion_length 100 --gpu_id 0 --replic_times 2 --name_ext R0 You could change the argument replic_times to get more generated motions. If you're testing the model you‘ve trained by you own, please replace the argument name with the name of checkpoint model you want to test.
You could use the argument --eval_type to generate new labels which hasn't seen in training stage.
- HumanAct12
python evaluate_motion_vae.py --name <Experiment_name> --dataset_type humanact12 --use_lie --time_counter --motion_length 60 --coarse_grained --gpu_id 0 --replic_times 5 --name_ext _R0 --eval_type <label>- NTU-RGBD
python evaluate_motion_vae.py --name <Experiment_name> --dataset_type ntu_rgbd_vibe --use_lie --time_counter --motion_length 60 --gpu_id 0 --replic_times 5 --name_ext R0 --eval_type <label>- CMU Mocap
python evaluate_motion_vae.py --name <Experiment_name> --dataset_type mocap --use_lie --time_counter --motion_length 60 --gpu_id 0 --replic_times 5 --name_ext R0 --eval_type <label>If you find this model or datasets useful for you research, please consider citing our work.
Contact Chuan Guo at cguo2 at ualberta.ca for any questions or comments
We have introduced vector mapping. Thus there should be arguments --lambda_sim and --lambda_tri. An example is given below:
python train_motion_vae.py --name mapping_humanact12_nolie --dataset_type humanact12 --batch_size 128 --motion_length 60 --coarse_grained --lambda_kld 0.001 --lambda_sim 0.0001 --lambda_tri 0.0001 --eval_every 2000 --plot_every 50 --print_every 20 --save_every 2000 --save_latest 50 --time_counter --gpu_id 0 --iters 50000Note that a large batch size will render it very very very slow!
Lie algebra is not supported.
Since large batch size would give rise to slow training, we limit the triplet loss to at most mapping_loss method, with default value 8).