• Python >=3.12.2
• Required packages: python install -r requirement.txt
• vicuna-7b-v1.3 from https://huggingface.co/lmsys/vicuna-7b-v1.3 in the folder 'LLMs' (or others such as Meta-llama3.2-8b-Instruct)
• CLIP: git clone https://github.com/openai/CLIP

This repository contains three experiments associated to three different tasks and datasets:

• Spoken text decoding (convers): Multimodal spoken text decoding during conversations (main task of this work).
• Perceived speech decoding (perceived): Decoding the textual content of listened stories.
• Brain captioning (NSD): Decoding the captions of viewed images using the NSD datasets.
• Decoding reading text from EEG signals.

In the following, we detail the steps to reproduce the results of each experiment presented in the paper.

• Update the configuration file by specifying the following paths: DATA_PATH (ex. data/convers), RAW_FMRI_DATA_PATH (ex. data/fmri_convers), MODELS_TRAIN_PATH (ex. trained_models/convers), LLM_PATH (ex. LLMs/Meta-llama3.2-8b-Instruct)
• Download the the Convers datasets version 2.2.0 from the OpenNeuro platform ds001740 inside RAW_FMRI_DATA_PATH specified in the config file.
• Create a folder named "raw_data/transcriptions" inside DATA_PATH and upload the raw Transcriptions from the Ortolang platform convers/v2 into it:

With DATA_PATH set to "data/convers", you should obtain a structure similar to this after data preprocessing:

data
└── convers
    ├── preprocessed_fmri_data
    │   └── fMRI_data_200
    ├── processed_data
    │   ├── fMRI_data_split
    │   ├── interlocutor_text_data
    │   └── participant_text_data
    ├── raw_data
    │   ├── transcriptions
    │   └── fmri
    ├── test.json
    └── train.json

# Preprocessing raw data
python exps/convers/process_raw_bold_signal.py --n_rois 200 # Parcellation using 200 ROIs
python exps/convers/data_builder_tools/split_bold_files.py  # Processing raw 4D voxel BOLD signals and segmenting them into fixed-duration chunks
python exps/convers/data_builder_tools/textgrid_to_text.py # Processing transcription files (conversations) and segmenting them into fixed-duration text sequences

# Building training and test data
python exps/convers/data_builder_tools/build_data.py # Using json files to save paths of bold chunks and the [input, output] text for instruction tuning
python exps/convers/data_builder_tools/build_tokenizer.py # Building the tokenizer for the first stage of training

# Training and testing after each save_epoch
python  exps/convers/train_stage1.py -m DeconvBipartiteTransformerConv --batch_size 128 --epochs 200 # Stage-1: training the DeconvBipartite Transformer
python  exps/convers/train_stage2.py --batch_size 32 --epochs 100  -m BrainDEC_V0  --save_epochs 50 # Stage-2. Note: BrainDEC_V1 or BrainDEC_V2 converge quickly than V0, only 20 epochs are needed.

# Evaluate the results of the test set and save the scores
python exps/convers/evaluation.py   

• Update the configuration file by specifying the following paths: RAW_FMRI_DATA_PATH (ex. data/perceived), MODELS_TRAIN_PATH (ex. trained_models/perceived), and LLM_PATH (ex. LLMs/Meta-llama3.2-8b-Instruct).
• In the folders "DATA_TRAIN_DIR" and "DATA_TEST_DIR" (see the config file), download the training and test datasets as outlined in this project semantic-decoding.

With DATA_PATH set to "data/perceived" for example, you should obtain a structure similar to this after data preprocessing:

data
└── perceived
    ├── data_test
    ├── data_train
    └── processed
        ├── S1
        ├── S2
        ├── S3
        ├── fMRI_data_test_split
        └── fMRI_data_train_split

# Data preparation
python exps/perceived/prepare_datasets.py -s $subject (for $subject  in ['S1', 'S2', 'S3'])

# Build tokenizer for stage 1
python exps/perceived/build_tokenizer.py

# Stage-1 training (in a cross-subject manner)
python exps/perceived/train_stage1.py --batch_size 128

# Stage-2 training (for each subject separately)
python exps/perceived/train_stage2.py --batch_size 32 -s $subject (for $subject  in ['S1', 'S2', 'S3'])

# Evaluate the results of the test set and save the scores
python exps/perceived/evaluation.py $subject (for $subject  in ['S1', 'S2', 'S3'])

This is a comparison with brain understanding benchmark (BrainHub), based on Natural Scenes Dataset NSD and COCO.

• The processed datasets are available in here.
• Download the datasets using this script.
• Download COCO annotations from this link in the folder 'tools'
• Update the configuration file to specify the paths, and eventually to modify the hyperparameters.

data
└── nsd
    ├── webdataset_avg_split
    │   ├── test
    │   ├── train
    └── └── val

• To train and evaluate the model:

# Stage-1 training (in a cross-subject manner)
python exps/zuco/build_tokenizer.py
python exps/nsd/train_stage1.py --batch_size 128

# Stage-2 training (for each subject separately)
python exps/nsd/train_stage2.py --epochs 6 --save_epochs 1 --batch_size 32 -s $subject (choices=[1, 2, 5, 7])

To get the evaluation scores for each subject based on the generated files of the test set, refer to the Benchmark project.

With DATA_PATH set to "data/nsd", you should obtain the following structure:

The results presented here improve upon those reported in the paper by (1) training the first stage in a cross-subject manner, (2) using curated COCO annotations (COCO_73k_annots_curated.npy) and (3) adjusting the decoder LLM’s inference hyperparameters (see the configuration file). Results may vary slightly due to initialization and non-deterministic algorithms, but the variation remains low. Reported BrainDEC values are averaged over three runs. We compare our model with existing methods from the BrainHub benchmark:

The same raw data and preprocessing presented in EEG-To-Text are employed here.

• Update the configuration files "configs/configs_zuco.py" by specifying the paths similarly to the previous experiments.
• Download the following folders from ZuCo v1.0 and place them in the DATA_PATH specified in the config file (e.g., data/zuco/task1-SR/Matlab_files, etc.).
• Download task1-NR/Matlab_files from ZuCo v2.0 and place it as task2-NR-2.0/Matlab_files inside DATA_PATH.
• Generate the preprocessed data using the following instructions:

With DATA_PATH set to data/zuco, for example, you should obtain the following structure after data preprocessing:

data
└── zuco
    ├── processed
    │   ├── task1-SR
    │   ├── task2-NR
    │   ├── task2-NR-2.0
    │   └── task3-TSR
    ├── task1-SR
    │   └── Matlab_files
    ├── task2-NR
    │   └── Matlab_files
    ├── task2-NR-2.0
    │   └── Matlab_files
    └── task3-TSR
        └── Matlab_files

# Data preparation
python exps/zuco/preprocess_data.py -t task1-SR
python exps/zuco/preprocess_data.py -t task2-NR
python exps/zuco/preprocess_data.py -t task3-TSR
python exps/zuco/preprocess_data_v2.py

# Build tokenizer for stage-1
python exps/zuco/build_tokenizer.py

# Training and evaluation
python exps/zuco/train_stage1.py --batch_size 128 --epochs 20
python exps/zuco/train_stage2.py --batch_size 16 --epochs 4
python exps/zuco/evaluation.py

• Apply the proposed methodology for NSD datasets.
• Test other LLM decoders.
• Add experiments for decoding text from EEG signals.
• Cross-subject training for NSD dataset.

• The structure of this repository is in work progress

• Some parts of the code of this project are adapted from InstructBlip, we thank the authors for their great work.

• In the comparison on perceived speech decoding, we used the same datasets and configuration setup in this article. Data preprocessing and preparation scripts are taken from this link. We thank the authors for their great work.

@article{hmamouche2026braindec103589,
title = {BrainDEC: A Multimodal LLM for the Non-Invasive Decoding of Text from Brain Recordings},
journal = {Information Fusion},
volume = {127},
pages = {103589},
year = {2026},
issn = {1566-2535},
doi = {https://doi.org/10.1016/j.inffus.2025.103589},
url = {https://www.sciencedirect.com/science/article/pii/S156625352500661X},
author = {Youssef Hmamouche and Ismail Chihab and Lahoucine Kdouri and Amal El Fallah Seghrouchni}
}