Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model

Official implementation of Ctrl-Adapter, an efficient and versatile framework that adds diverse controls to any image/video diffusion models by adapting pretrained ControlNets.

Han Lin, Jaemin Cho, Abhay Zala, Mohit Bansal

CTRL-Adapter is an efficient and versatile framework for adding diverse spatial controls to any image or video diffusion model. It supports a variety of useful applications, including video control, video control with multiple conditions, video control with sparse frame conditions, image control, zero-shot transfer to unseen conditions, and video editing.

🔧 Setup

Environment Setup

To make our codebase easy to use, the primary libraries that need to be installed are Torch, Diffusers, and Transformers. Specific versions of these libraries are not required; the default versions should work fine :)

conda create -n ctrl-adapter python=3.10
conda activate ctrl-adapter
pip install -r requirements.txt

🔮 Inference

Video Generation with Condition Control

Here is a sample script that utilizes I2VGen-XL as the backbone model, with a depth map as the control condition. Inference consumes approximately 22GB of GPU memory on a single RTX 4090 GPU. The amount of memory required for inference depends on the backbone model used.

sh scripts/depth_ctrladapter_inference.sh

📝 TODO List

• Release environment setup, inference code for I2VGen-XL, and model checkpoints. (WIP)
• Release checkpoints and inference code for other models. (WIP)
• Release training code, and guideline to adapt our Ctrl-Adapter to new image/video diffusion models.
• Release evaluation code.

💗 Please let us know in the issues or PRs if you're interested in any relevant backbones or down-stream tasks that can be implemented by our Ctrl-Adapter framework!

📚 BibTeX

If you find our project useful in your research, please cite the following paper:

@misc{lin2024ctrladapter,
      title={Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model}, 
      author={Han Lin and Jaemin Cho and Abhay Zala and Mohit Bansal},
      year={2024},
      eprint={2404.09967},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

🙏 Acknowledgements

The development of Ctrl-Adapter has been greatly inspired by the following amazing works and teams:

• SDXL
• I2VGen-XL
• Hotshot-XL
• Stable Video Diffusion

We hope that releasing this model/codebase helps the community to continue pushing these creative tools forward in an open and responsible way.