This repository contains the official source code and data for our paper:
Future Transformer for Long-term Action Anticipation
Dayoung Gong, Joonseok Lee, Manjin Kim, Seong Jong Ha, and Minsu Cho POSTECH & NCSOFT CVPR, New Orleans, 2022.
If you find our code or paper useful, please consider citing our paper:
@inproceedings{gong2022future,
title={Future Transformer for Long-term Action Anticipation},
author={Gong, Dayoung and Lee, Joonseok and Kim, Manjin and Ha, Seong Jong and Cho, Minsu},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={3052--3061},
year={2022}
}We conduct experiments on Breakfast with 4 splits and 50Salads with 5 splits.
| Dataset | Model | obs 0.2, pred 0.1 | obs 0.2, pred 0.2 | obs 0.2, pred 0.3 | obs 0.2, pred 0.4 | obs 0.3, pred 0.1 | obs 0.3, pred 0.2 | obs 0.3, pred 0.3 | obs 0.3, pred 0.4 | Checkpoint (Splits) |
|---|---|---|---|---|---|---|---|---|---|---|
| Breakfast | FUTR | 27.71 | 24.56 | 22.84 | 22.05 | 32.27 | 29.89 | 27.49 | 25.88 | 1 2 3 4 |
| 50Salads | FUTR | 37.01 | 27.81 | 22.46 | 16.75 | 33.32 | 23.17 | 22.14 | 15.49 | 1 2 3 4 5 |
- Conda environment settings:
conda env export > futr.yaml
conda activate futrDownload the data from https://mega.nz/file/O6wXlSTS#wcEoDT4Ctq5HRq_hV-aWeVF1_JB3cacQBQqOLjCIbc8 .
Create a directory './datasets' for the two datasets and place each dataset to have following directory structure:
../ # parent directory
├── ./ # current (project) directory
│ ├── data/ # (dir.) dataloaders for action anticipation dataset
│ ├── model/ # (dir.) implementation of Hypercorrelation Squeeze Network model
│ ├── README.md # intstruction for reproduction
│ ├── train.py # code for training FUTR
│ ├── predict.py # code for testing FUTR
│ ├── otps.py # code for arguments
│ └── utils.py # code for helper functions
└── datasets/
├── breakfast/ # Breakfast dataset
│ ├── groundTruth/
│ ├── features/
│ ├── mapping.txt
│ └── ...
├── 50salads/ # 50salads dataset
│ ├── groundTruth/
│ ├── features/
│ ├── mapping.txt
│ └── ..../scripts/train.sh $split_num
./scripts/50s_train.sh $split_num
./scripts/predict.sh $split_num
./scripts/50s_predict.sh $split_num
We thank Yazan Abu Farha for providing the code of Long-term anticipation of activities with cycle consistency and for helping us to reproduce experiments.