[AAAI 2025] A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter

TL;DR: Wander enables efficient transfer learning for multimodal models with fine-grained interactions between multimodal sequences.

Illustration of the main component of Wander, low-rank sequence fusion.

You can choose to add Wander to the last layer of the pre-trained model before the prediction head or every intermediate layer of the Transformer.

The Wander adapter is implemented in the models/adapter.py file. Due to the simplicity of our task, we only use the last transformer layer of the pre-trained model as the input of the Wander adapter. An example of fine-tuning pre-trained models with Wander on MOSI:

python run/mosi_run..py --pretrained_model [pretrained_model_name] --rank [rank] --drank [drank] --trank [trank]

If you want to use Wander in your own model, you can refer to the models/adapter.py file. You need a pre-trained multimodal model first before adding Wander. If your pre-trained model is large, you can consider adding Wander to every intermediate layer of the Transformer. Here is a simple example of adding Wander to every intermediate layer of the Transformer:

class PretrainedModel(nn.Module):
    def __init__(self, ...):
        super(PretrainedModel, self).__init__()
        self.tflayer1 = TransformerLayer()
        ...
        self.tflayer2 = TransformerLayer()
        ...
        self.head = nn.Linear(hidden_size, num_classes)
        ...
    
    def forward(self, x):
        ...


class WanderModel(PretrainedModel):
    def __init__(self, ...):
        super(WanderModel, self).__init__()
        self.wander_layer1 = Adapter()
        ...
        self.wander_layer2 = Adapter()
        ...
    
    def forward(self, x):
        x = self.tflayer1(x)
        x = self.wander_layer1(x)
        ...
        x = self.tflayer2(x)
        x = self.wander_layer2(x)
        ...
        x = self.head(x)
        ...
        return x

If you find this work useful, please cite our paper:

@article{guo2025wander,
  title={A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter},
  author={Guo, Zirun and Cheng, Xize and Wu, Yangyang and Jin, Tao},
  journal={The 39th Annual AAAI Conference on Artificial Intelligence},
  year={2025}
}