This repository provides the official PyTorch implementation for the following paper:

ReliTalk: Relightable Talking Portrait Generation from a Single Video
Haonan Qiu, Zhaoxi Chen, Yuming Jiang, Hang Zhou, Wayne Wu, Xiangyu Fan, Lei Yang, and Ziwei Liu

From MMLab@NTU affiliated with S-Lab, Nanyang Technological University and SenseTime Research.

Video Data: HDTF

• Clone this repo: git clone --recursive [email protected]:arthur-qiu/ReliTalk.git
• Create a conda environment conda env create -f environment.yml and activate conda activate ReliTalk
• We use libmise to extract 3D meshes, build libmise by running cd code; python setup.py install
• Download FLAME model, choose FLAME 2020 and unzip it, copy 'generic_model.pkl' into ./code/flame/FLAME2020

Prepare the dataset following intructions in ./preprocess/README.md.

Link the dataset folder to ./data/datasets. Link the experiment output folder to ./data/experiments.

cd code
python scripts/exp_runner.py --conf ./confs/IMavatar_supervised_Obama.conf

python scripts/exp_runner.py --conf ./confs/IMavatar_supervised_Obama_test.conf --is_eval --checkpoint [epoch1]

mv ../data/experiments/Obama/IMavatar/Obama_train/eval/Obama_eval/epoch_[epoch1]/normal ../data/datasets/Obama/Obama/Obama_eval/

mv ../data/experiments/Obama/IMavatar/Obama_train/eval/Obama_train/epoch_[epoch1]/normal ../data/datasets/Obama/Obama/Obama_train/

mv ../data/experiments/Obama ../data/experiments/Obama_store

python scripts/exp_runner_relight.py --conf ./confs/IMavatar_supervised_Obama_light.conf

python scripts/exp_runner_relight.py --conf ./confs/IMavatar_supervised_Obama_test_light.conf --is_eval --checkpoint [epoch2]

If you find this work useful for your research, please consider citing our paper:

@misc{qiu2023relitalk,
      title={ReliTalk: Relightable Talking Portrait Generation from a Single Video}, 
      author={Haonan Qiu and Zhaoxi Chen and Yuming Jiang and Hang Zhou and Xiangyu Fan and Lei Yang and Wayne Wu and Ziwei Liu},
      year={2023},
      eprint={2309.02434},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

This code borrows heavily from IMavatar.