This is the official implementation of MindCross: Fast New Subject Adaptation with Limited Data for Cross-subject Video Reconstruction from Brain Signals. (Accepted by AAAI 2026)
You can also see the poster to breifly know our work.
Xuan-Hao Liu, Yan-Kai Liu, Tian-Yi Zhou, Bao-Liang Lu, Wei-Long Zheng*
Shanghai Jiao Tong University
We present, MindCross, a novel cross-subject brain decoding framework for rapid adaptation to new subjects: 1) Shared-specific Encoder Architecture MindCross learns subject-invariant information by a shared encoder and allocates each subject a specific encoder to learn subject-related information. 2) Fast New Subject Calibration MindCross can rapidly adapt to him/her by only updating the parameter of the new subject while other modules are all frozen. 3) Collaboration Decoding When the new subject has limited training data, MindCross decodes the semantic embeddings not only from the specific encoder of the new subject, but also from the encoders of Top K existing subjects similar to the new subject.
Discussed in paper, our framework is used for brain data - semantic latent alignment, and basically it can be used for any downstream T2V generative models. Thus, we only give the code of our framework, not the code of taming T2V models.
The processed CC2017 dataset can be downloaded from CC2017 link. (Thanks the effort of NeuroClips team)
The SEED-DV dataset can be applied from SEED-DV link.
You should extract your own caption_emdddings by the T2V model you used.
Training
python train_model_cc2017.py
python train_model_seeddv.py
Calibration
python train_calib_model_cc2017.py
python train_calib_model_seeddv.py
Inference
python inference_model_cc2017.py
python inference_model_seeddv.py
If you find this work useful in your research, please consider citing:
@inproceedings{liu2026mindcross,
title={MindCross: Fast New Subject Adaptation with Limited Data for Cross-subject Video Reconstruction from Brain Signals},
author={Liu, Xuan-Hao and Liu, Yan-Kai and Zhou, Tian-Yi and Lu, Bao-Liang and Zheng, Wei-Long},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
year={2026}
}