This is the implementations for PIG-Nav: Key Insights for Pretrained Image-Goal Navigation Models.
- We propose PIG-Nav, a pretrained model for image-goal navigation with diverse indoor and outdoor navigation datasets, featuring an early-fusion network architecture with appropriately pretrained Vision Transformer encoder, and high-level auxiliary tasks designed for navigation agents.
- On two large 3D game environments (Highrise and Sanctuary), PIG-Nav achieves an average improvement of 22.6% in zero-shot setting and 37.5% improvement in finetuning setting over existing visual navigation models.
| Model Name | HF Path | Note |
|---|---|---|
| PIG-Nav | PIG-Nav | The pretrained PIG-Nav model with diverse navigation dataset. |
| PIG-Nav (w/o early-fuse) | PIG-Nav-NoEarlyfuse | The pretrained PIG-Nav model without early fuse architecture. |
| PIG-Nav (w/o aux) | PIG-Nav-NoAux | The pretrained PIG-Nav model without auxiliary tasks. |
| PIG-Nav-Highrise-finetuned | PIG-Nav-Finetuned-Highrise | The PIG-Nav model fine-tuned on Highrise environment. |
| PIG-Nav-Highrise-finetuned | PIG-Nav-Finetuned-Sanctuary | The PIG-Nav model fine-tuned on Sanctuary environment. |
Use uv sync for installing the environment, and use source .venv/bin/activate to activate the environment.
Pretraining datasets: available at https://huggingface.co/datasets/zpschang/PIG-Nav-Dataset-Pretrain.
Use cat pignav_pretrain_dataset.tar.part-* | tar -xvf - for extracting the pretraining dataset.
Finetuning datasets for Highrise and Sanctuary: available at https://huggingface.co/datasets/zpschang/PIG-Nav-Dataset-Finetune.
Run bash scripts/pretrain.sh for pretraining PIG-Nav. The config file for pretraining is in configs/pretrain.yaml. Please modify the data_folder and data_split_folder to your pretraining dataset path. The pretrained model checkpoints will be saved under log_dir in scripts/pretrain.sh.
For the full configuration information for PIG-Nav training, see model_config.md.
Run bash scripts/finetune_[env].sh for finetuning PIG-Nav on Highrise and Sanctuary game environments. The config file for finetuning is in configs/finetune_[env].yaml. Please modify the data_folder and data_split_folder to your finetuning dataset path.
For the full configuration information for PIG-Nav finetuning, see model_config.md.
First, download the ShooterGame environment. To start the environment under Linux, enter Linux/ path, and run ./ShooterGame.sh [port_number] to start the environment with the port number.
Run bash scripts/eval_highrise.sh or bash scripts/eval_sanctuary.sh to start evaluate the PIG-Nav under two environments. You should modify the port value to match the port number for starting the environment. The success rates will be saved in logs/, and the video demos will be saved in videos/.