Retina : Low-Power Eye Tracking with Event Camera and Spiking Hardware
🧑🏻🚀 Pietro Bonazzi 1,
Sizhen Bian 1,
Giovanni Lippolis 2,
Yawei Li1,
Sadique Sheik 2,
Michele Magno1
1 ETH Zurich, Switzerland
2 SynSense AG, Switzerland
In the following GIFs, Yellow represents the Ground Truth (GT), and Green represents the Prediction. These images are taken from the validation set.
Leave a star to support our open source initiative!⭐️
@InProceedings{Bonazzi_2024_CVPR,
author = {Bonazzi, Pietro and Bian, Sizhen and Lippolis, Giovanni and Li, Yawei and Sheik, Sadique and Magno, Michele},
title = {Retina : Low-Power Eye Tracking with Event Camera and Spiking Hardware},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2024},
pages = {5684-5692}
}
git clone https://github.com/pbonazzi/retina.git
cd retina
conda create -n retina python=3.8 -y
conda activate retina
pip install -r requirements.txt
Click here to download the Ini-30 Dataset.
Verify the structure:
.
├── name
│ ├── annotations.csv
│ └── events.aedat4
├── ...
├── silver.csv
I have benchmarked different methods to aggregate event data and the np.add.at operators was the fastest. The dv.Accumulator in data/datasets/ini_30/ini_30_aeadat_processsor.py is currently the bottleneck causing the dataloader in the Ini-30 Dataset to be slow. You can speed it up by saving/loading the event windows.
Follow the instruction here to download the 3ET Dataset
Rename .env.example to .env and change its INI30_DATA_PATH and 3ET_DATA_PATH.
Click here to download a pretrained model.
See the list of arguments in the launch_fire function and the config/defaults.yaml.
python3 -m scripts.train --run_name="retina-ann"
Install dependencies for TFlite
pip install onnx2tf onnx-tf tensorflow onnx_graphsurgeon tf_keras
Quantiazation INT8 : Example for 3et_on_ini30
python3 -m onnxsim output/retina-ann-v6-evs-1000/models/model.onnx output/retina-ann-v6-evs-1000/models/model-simplified.onnx
onnx2tf -i output/retina-ann-v6-evs-1000/models/model.onnx -o output/retina-ann-v6-evs-1000/models/model_tf
python3 -m scripts.quantize