OPUS: Occupancy Prediction Using a Saprse Set

• Authors: Jiabao Wang*, Zhaojiang Liu*, Qiang Meng, Liujiang Yan, Ke Wang, Jie Yang, Wei Liu, Qibin Hou#, Ming-Ming Cheng
  (* Equal contribition, # Corresponding author)
• Paper in arXiv | 知乎

• [2025/01/10]: We release the visualization code.
• [2024/09/26]: OPUS is accepeted by NeurIPS 2024 🚀 🚀 🚀 .
• [2024/09/17]: We release an initial version of OPUS. It achieves promising performance of 41.2 RayIoU and 36.2 mIoU on the NuScene-Occ3D dataset.

Occupancy prediction, aiming at predicting the occupancy status within voxelized 3D environment, is quickly gaining momentum within the autonomous driving community. Mainstream occupancy prediction works first discretize the 3D environment into voxels, then perform classification on such dense grids. However, inspection on sample data reveals that the vast majority of voxels is unoccupied. Performing classification on these empty voxels demands suboptimal computation resource allocation, and reducing such empty voxels necessitates complex algorithm designs. To this end, we present a novel perspective on the occupancy prediction task: formulating it as a streamlined set prediction paradigm without the need for explicit space modeling or complex sparsification procedures. Our proposed framework, called OPUS, utilizes a transformer encoder-decoder architecture to simultaneously predict occupied locations and classes using a set of learnable queries. Firstly, we employ the Chamfer distance loss to scale the set-to-set comparison problem to unprecedented magnitudes, making training such model end-to-end a reality. Subsequently, semantic classes are adaptively assigned using nearest neighbor search based on the learned locations. In addition, OPUS incorporates a suite of non-trivial strategies to enhance model performance, including coarse-to-fine learning, consistent point sampling, and adaptive re-weighting, etc. Finally, compared with current state-of-the-art methods, our lightest model achieves superior RayIoU on the Occ3D-nuScenes dataset at near $2\times$ FPS, while our heaviest model surpasses previous best results by 6.1 RayIoU.

note: Q denotes query numbers. P is the number of predicted points per query.

直接使用 SparseOcc 的环境（PyTorch 2.0 + CUDA 11.8）即可进行训练与推理

We build OPUS based on Pytorch 1.13.1 + CUDA 11.6

conda create -n opus python=3.8
conda activate opus
conda install pytorch==1.13.1 torchvision==0.14.1 pytorch-cuda=11.6 -c pytorch -c nvidia

Install other dependencies:

pip install openmim
mim install mmcv-full==1.6.0
mim install mmdet==2.28.2
mim install mmsegmentation==0.30.0
mim install mmdet3d==1.0.0rc6
pip install fvcore einops

Install turbojpeg and pillow-simd to speed up data loading (optional but important):

sudo apt-get update
sudo apt-get install -y libturbojpeg
pip install pyturbojpeg
pip uninstall pillow
pip install pillow-simd==9.0.0.post1
pip install fvcore  # 再运行一遍，把 pillow 装回去

Compile CUDA extensions:

cd models/csrc
python setup.py build_ext --inplace

不能直接使用 SparseOcc 提供的 pkl 文件，需要手动生成 第 3 步耗时较长，需要若干小时

1. Download nuScenes from https://www.nuscenes.org/nuscenes and place it in folder data/nuscenes.

2. Download Occ3d-nuScenes from https://tsinghua-mars-lab.github.io/Occ3D/ and place it in data/nuscenes/gts

3. Prepare data with scripts provided by mmdet3d:

mim run mmdet3d create_data nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --extra-tag nuscenes

4. Perform data preparation for OPUS:

python gen_sweep_info.py

The final folder structure would be

data/nuscenes
├── maps
├── nuscenes_infos_test_sweep.pkl
├── nuscenes_infos_train_sweep.pkl
├── nuscenes_infos_train_mini_sweep.pkl
├── nuscenes_infos_val_sweep.pkl
├── nuscenes_infos_val_mini_sweep.pkl
├── samples
├── sweeps
├── v1.0-test
└── v1.0-trainval

Note: These *.pkl files can also be generated with our script: gen_sweep_info.py.

Download pre-trained weights provided by mmdet3d, and put them in directory pretrain/:

pretrain
├── cascade_mask_rcnn_r101_fpn_1x_nuim_20201024_134804-45215b1e.pth
├── cascade_mask_rcnn_r50_fpn_coco-20e_20e_nuim_20201009_124951-40963960.pth

Train OPUS with a single GPU:

python train.py --config configs/opus-t_r50_704x256_8f_24e.py

Train OPUS with 8 GPUs:

bash dist_train.sh 8 configs/opus-t_r50_704x256_8f_24e.py

Note: The batch size for each GPU will be scaled automatically. So there is no need to modify the batch_size in configurations.

Single-GPU evaluation:

export CUDA_VISIBLE_DEVICES=0
python val.py --config configs/opus-t_r50_704x256_8f_100e.py --weights checkpoints/opus-t_r50_nusc-occ3d_100e.pth

Multi-GPU evaluation:

export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
torchrun --nproc_per_node 8 val.py --config configs/opus-t_r50_704x256_8f_100e.py --weights checkpoints/opus-t_r50_nusc-occ3d_100e.pth

Visualizing results

python visualize.py --config configs/opus-t_r50_704x256_8f_100e.py --weights checkpoints/opus-t_r50_nusc-occ3d_100e.pth

Visualizing inputs and ground-truths

python visualize.py --config configs/opus-t_r50_704x256_8f_100e.py --weights checkpoints/opus-t_r50_nusc-occ3d_100e.pth --vis-input --vis-gt

If this work is helpful for your research, please consider citing the following entry.

@inproceedings{wang2024opus,
  title={Opus: occupancy prediction using a sparse set},
  author={Wang, Jiabao and Liu, Zhaojiang and Meng, Qiang and Yan, Liujiang and Wang, Ke and Yang, Jie and Liu, Wei and Hou, Qibin and Cheng, Mingming}
  booktitle={Advances in Neural Information Processing Systems},
  year={2024}
}

Our code is developed on top of following open source codebase:

• SparseBEV
• SparseOcc

We sincerely appreciate their amazing works.