Blender Auto Poser

A Blender addon that uses AI to predict poses from sparse inputs.

Based on ProtoRes: Proto-Residual Network for Pose Authoring via Learned Inverse Kinematics.
Original paper by Boris N. Oreshkin, Florent Bocquelet, Félix G. Harvey, Bay Raitt and Dominic Laflamme.

@inproceedings{oreshkin2022protores:,
  title={ProtoRes: Proto-Residual Network for Pose Authoring via Learned Inverse Kinematics},
  author={Boris N. Oreshkin and Florent Bocquelet and F{\'{e}}lix G. Harvey and Bay Raitt and Dominic Laflamme},
  booktitle={International Conference on Learning Representations},
  year={2022}
}

Iterated with Pose and Skeleton-aware Neural IK for Pose and Motion Editing.
Original paper by Dhruv Agrawal, Martin Guay, Jakob Buhmann, Dominik Borer, and Robert W. Sumner.

@inproceedings{agrawal2023skel,
  author = {Agrawal, Dhruv and Guay, Martin and Buhmann, Jakob and Borer, Dominik and Sumner, Robert W.},
  title = {Pose and Skeleton-Aware Neural IK for Pose and Motion Editing},
  year = {2023},
  isbn = {9798400703157},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3610548.3618217},
  doi = {10.1145/3610548.3618217},
  booktitle = {SIGGRAPH Asia 2023 Conference Papers},
  articleno = {49},
  numpages = {10},
  location = {Sydney, NSW, Australia},
  series = {SA '23}
}

Download Trained Model

Place the *.ckpt file in the auto_poser/poser/models folder.
https://drive.google.com/file/d/15nRp_vhEKewYohcQVxa9jjjPZxqoXIYi/view?usp=drive_link

Training

This is not needed to use the addon, but if you want to train the model yourself, you can follow the instructions below.

Environment Setup

# Python version (Blender uses Python 3.11)
Python 3.11.11

# Install dependencies (Virtual Environment is recommended)
# Change the Cuda version to your GPU
pip install -r requirements.txt

# Confirm that torch was installed
python -c "import torch; print(f'PyTorch version: {torch.__version__}'); print(f'CUDA available: {torch.cuda.is_available()}'); print(f'CUDA version: {torch.version.cuda}'); print(f'GPU device name: {torch.cuda.get_device_name(0)}' if torch.cuda.is_available() else 'No GPU available')"

Visual Studio Code

I recommend using Visual Studio Code with the extension Blender Development by Jacques Lucke.
Inside "User Settings (JSON)" add the following with your specific paths:

"blender.additionalArguments": ["E:\\Blender\\projects\\auto_poser.blend"],
"blender.executables": [
  {
    "path": "C:\\Program Files\\Blender Foundation\\Blender 4.4\\blender.exe",
    "name": "",
    "isDebug": false
  }
]

And then using command palette (Ctrl+Shift+P) you can run with Blender: Start and use Blender: Reload Addons to reload the addon if you make any changes.

Dataset

The dataset I used was created with mixamo animations. To download animations from it to train, you have to:

• Select the Y Bot character (if you want a different character you have to change constants.py), search for "packs", download them and place the folders inside auto_poser/data/animations.
  
• Inside Blender, open the Auto Poser's sidepanel, go to settings and click Parse Animations.
  
• This will take a while, I recommend opening Blender's console to see the progress before starting, or if you are using Blender Development by Jacques Lucke you can see the progress in VSCode's terminal.
  

Training the Model

# Start training
python -m poser.trainer

# Or redirect output to log file
python -m poser.trainer > training_output.log 2>&1

Monitoring

# Launch TensorBoard
tensorboard --logdir=poser\models

# Open TensorBoard in a browser
http://localhost:6006/
http://127.0.0.1:6006/