Project Page · Huggingface Leaderboard· Paper@ArXiv· ECTV Dataset· Prompt list

Welcome to EvalCrafter, a comprehensive evaluation toolkit for AI-generated videos. Our innovative framework assesses generative models across visual, content, and motion qualities using 17 objective metrics and subjective user opinions, providing a reliable ranking for state-of-the-art text-to-video generation models. Dive into the world of unified and user-centric evaluation with EvalCrafter! 🚀🌍📊

Clone the repository:

git clone https://github.com/evalcrafter/EvalCrafter
cd EvalCrafter

Generate videos of your model using the 700 prompts provided in prompt700.txt or ./prompts and organize them in the following structure:

/EvalCrafter/videos
├── 0000.mp4
├── 0001.mp4
├── 0002.mp4
├── 0003.mp4
├── 0004.mp4
...
└── 0699.mp4

Please download all checkpoints using

cd checkpoints
bash download.sh

Alternatively, you can follow ./checkpoints/README.md to download pretrained models for specific metrics.

Note: Please organize the pretrained models in this structure:

/EvalCrafter/checkpoints/
├── bert-base-uncased
├── blip2-opt-2.7b
├── ckpt
├── clip-vit-base-patch32
├── Dover
├── FlowNet2_checkpoint.pth.tar
├── pt_inception-2015-12-05-6726825d.pth
├── RAFT
├── stable-diffusion-xl-base-1.0
├── tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb_20220906-2692d16c.pth
├── vgg_face_weights.h5
└── VideoMAE

docker pull bruceliu1/evalcrafter:v1

1. Run with command line:

   docker run --runtime=nvidia -it --shm-size "15G" -v $EC_path:$EC_path bruceliu1/evalcrafter:v1 \
      bash -c "source /opt/conda/bin/activate EvalCrafter \
         && bash $bash_file $EC_path $EC_path/videos"
   

   🔁 Please replace $EC_path, $bash_file, and $dir_videos with your local path to EvalCrafter, EvalCrafter/start.sh, and EvalCrafter/videos, respectively.

Alternatively, you can:

2. Enter the Docker container and run:

   docker run --runtime=nvidia -it --shm-size "15G" -v $EC_path:$EC_path bruceliu1/evalcrafter:v1 bash
   cd $EC_path
   bash start.sh $EC_path $dir_videos
   

🔧 To test a specific metric, pick out the code for the metric in start.sh. For example, to test the Celebrity ID Score:

docker run --runtime=nvidia -it --shm-size "15G" -v $EC_path:$EC_path bruceliu1/evalcrafter:v1 bash
cd $EC_path
cd /metrics/deepface
python3 celebrity_id_score.py --dir_videos $dir_videos

This work is based on the following open-source repositories:

• deepface
• DOVER
• mmaction2
• CLIP
• RAFT
• pytorch-gan-metrics
• SDXL
• PaddleOCR
• SAM-Track
• BILIP2
• HRS-Bench
• fast_blind_video_consistency

If you find this repository helpful, please consider citing it in your research:

@article{liu2023evalcrafter,
title={Evalcrafter: Benchmarking and evaluating large video generation models},
author={Liu, Yaofang and Cun, Xiaodong and Liu, Xuebo and Wang, Xintao and Zhang, Yong and Chen, Haoxin and Liu, Yang and Zeng, Tieyong and Chan, Raymond and Shan, Ying},
journal={arXiv preprint arXiv:2310.11440},
year={2023}
}

• VideoCrafter1: Open Diffusion Models for High-Quality Video Generation
• ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models
• TaleCrafter: Interactive Story Visualization with Multiple Characters