This is a forked repository of XVFI (eXtreme Video Frame Interpolation)

This repository provides a pruned version of the original source code, designed exclusively to execute frame interpolation in custom videos with pretrained models. Its primary function is to easily increase the FPS of the input video with an off-the-shelf video interpolation method. The choice of XVFI (ICCV2021) is motivated by its compelling visual results on custom videos. Notably, training-related components and metric performance utilities are excluded from this repository. Additionally, the code is reorganized as a python package that can be added as a submodule in other projects.

1. Installation
2. Usage
3. Acknowledgements
4. License

The code is implemented using PyTorch 1.9.0 and was tested on Ubuntu 20.04 machine (Python 3.8, CUDA 11.3).

1. Clone repository

   git clone https://github.com/germanftv/XVFI
   cd XVFI

2. Create conda environment

   conda env create -n XVFI -f environment.yml

3. Install pip dependencies

   pip install -r pip_requirements.txt
   cd ..

4. Download pretrained models and place them as indicated:

   • X4K1000FPS, which was trained on X-TRAIN dataset must be located in ./XVFI/checkpoint_dir/XVFInet_X4K1000FPS_exp1

     XVFI
     └── checkpoint_dir
         └── XVFInet_X4K1000FPS_exp1
             └── XVFInet_X4K1000FPS_exp1_latest.pt           
     

   • Vimeo, which was trained on Vimeo90K dataset must be located in ./XVFI/checkpoint_dir/XVFInet_Vimeo_exp1

     XVFI
     └── checkpoint_dir
         └── XVFInet_Vimeo_exp1
             └── XVFInet_Vimeo_exp1_latest.pt           
     

1. Extract the frames from your input video. With ffmpeg, you can use the following command:

   ffmpeg -i $INPUT_VIDEO -start_number 0 -q:v 0 $INPUT_DIR/%08d.png 

   where:

   • INPUT_VIDEO is the path to your input video.
   • INPUT_DIR is the path to the directory that contains the extracted frames.

2. Run XVFI either by using the Command Line Interface (CLI) or importing the python package in your python script:

   CLI:

   python -m XVFI.main --input_dir $INPUT_DIR --output_dir $OUTPUT_DIR --multiple $VI_FACTOR --pretrained $PRETRAINED --gpu $GPU --config $CONFIG

   Python script:

   import XVFI
   import argparse
   
   # Create an argument parser
   parser = argparse.ArgumentParser()
   args = parser.parse_args()
   
   # Declare required inputs
   args.input_dir = INPUT_DIR
   args.output_dir = OUTPUT_DIR
   args.multiple = VI_FACTOR
   # Optional:
   args.pretrained = PRETRAINED
   args.gpu = GPU
   args.config = CONFIG
   
   # Add default arguments for XVFI
   args = XVFI.add_default_args(args, parser)
   
   # Run XVFI
   XVFI.run(args)

   where:

   • INPUT_DIR is the path to the directory that contains the input frames.

   • OUTPUT_DIR is the path to the directory that contains the frames for the interpolated video.

   • VI_FACTOR is the video interpolation factor. For instance, 8 results in a video with x8 FPS.

   • PRETRAINED refers to the pretrained weights to be used. Choices: 'X4K1000FPS', 'Vimeo'. Default: 'X4K1000FPS'.

   • GPU is an integer id to a gpu device. Default: 0.

   • CONFIG is the path to the config file. Default: './XVFI/configs/default.yaml'.

     NOTE: In the config file, you can modify the lowest scale depth by regulating 'S_tst'. By default, we use the test settings that are used in the original source code. The reader is refered to the paper for more details about this hyper-parameter.

3. (Optional) Convert frames to video with ffmpeg using the command:

   ffmpeg -framerate 30 -pattern_type glob -i "$OUTPUT_DIR/*.png" -c:v libx264 -crf 1 -pix_fmt yuv420p $OUTPUT_VIDEO 

   where:

   • OUTPUT_DIR is the path to the directory that contains the frames for the interpolated video.
   • OUTPUT_VIDEO is the path interpolated video file with .mp4 extension.

Kudos to the original contributors of XVFI: Jihyong Oh, Hyeonjun Sim, and the Video & Image Computing Lab at KAIST, who provided a pytorch implementation of the paper.

The source code can be used for research and education only as stated by the original authors.