This repository contains the official implementation of UniPose. UniPose is a multi-modal LLM to understand, generate and edit 3D human poses. It enables seamless transformation of human poses across multi-modalities (visual, text, SMPL) and multi-tasks (comprehension, generation, editing).

1. Clone this repo and navigate to UniPose folder

   git clone https://github.com/liyiheng23/UniPose.git
   cd UniPose

2. Create Conda Environment and Install Package

   conda create -n unipose python=3.10 -y
   conda activate unipose
   pip3 install -r requirements.txt

3. Download Pre-trained Models and put these models under ./cache folder.

   Base Model:

   • CLIP-ViT: openai/clip-vit-large-patch14-336
   • LLaVA: liuhaotian/llava-v1.6-mistral-7b
   • smpl_models: smplh

   Ours provided Model:

   • unipose: L-yiheng/UniPose
   • tokenhmr_model: tokenhmr_model
   • pose-vqvae: pose-vqvae

   Now under ./cache, you should be able to see the following:

   cache
   ├── clip-vit-large-patch14-336
   ├── llava-v1.6-mistral-7b
   ├── pose_vqvae
   │   └── best_MPJPE.ckpt
   ├── smpl_models
   │   └── smplh
   │       ├── SMPLH_FEMALE.npz
   │       ├── SMPLH_MALE.npz
   │       └── SMPLH_NEUTRAL.npz
   ├── tokenhmr_model.ckpt
   └── unipose

python inference.py \
    --model-path cache/unipose \
    --model-base cache/llava-v1.6-mistral-7b \
    --config configs/inference.py

The provided code and pre-trained weights are licensed under the Apache 2.0 license.

This code is based on ChatPose, LISA, LLaVA and PoseScript. Some code are brought from TokenHMR, 4D-Humans. We thank the authors for their awesome work.

If you have any questions, please feel free to reach me out at [email protected].

If you find our work useful for your research, please consider citing our paper:

@article{li2024unipose,
  title={UniPose: A Unified Multimodal Framework for Human Pose Comprehension, Generation and Editing},
  author={Li, Yiheng and Hou, Ruibing and Chang, Hong and Shan, Shiguang and Chen, Xilin},
  journal={arXiv preprint arXiv:2411.16781},
  year={2024}
}