BEVBlip is an efficient and lightweight Vision-Language Model (VLM) based on BLIP architecture, trained for comprehensive Visual Question Answering (VQA) task introduced by DriveLM on nuScenes dataset. As the core idea, BEVBlip employs spatio-temporal Bird’s Eye View (BEV) maps acquired via BEVFormer as visual features and integrates visual and language features for enhanced traffic environment understanding. In order to align BEV features with language, a pre-training stage utilizing GPT generated data is executed.

High-level outline of the proposed approach:

The architecture of the pre-training model:

The bottom section illustrates offline data generation steps using BEVFormer and GPT-3.5. The upper right section shows the unified multimodal encoder-decoder with pretrained weights from BLIP. The upper left section depicts the compact vision transformer architecture, trained from scratch with BEV feature maps.

The architecture of the VQA model used for the fine-tuning on DriveLM task:

Left section shows the vision transformer, initialized with the weights from the pre-training stage. Right section illustrates the reconfiguration of text encoder and text decoder as question encoder and answer decoder respectively.

Sources and references:

• BLIP: https://github.com/salesforce/BLIP
• BEVFormer: https://github.com/fundamentalvision/BEVFormer
• DriveLM: https://github.com/OpenDriveLab/DriveLM