Challenge: The interpretation and analysis of Magnetic Resonance Imaging scans in clinical AI systems rely on accurate understanding of image contrast. While contrast is determined by acquisition parameters stored in DICOM metadata, real-world clinical datasets often suffer from noisy, incomplete, or inconsistent metadata. Broad labels like 'T1 weighted' or 'T2-weighted' are commonly used, but offer only coarse and insufficient descriptions.
Problem: In many real-world clinical datasets, such labels are missing altogether, leaving raw acquisition parameters, such as echo time and repetition time, as the only available contrast indicators. These parameters directly govern image appearance and are critical for accurate interpretation, reliable retrieval, and integration into clinical workflows.
Solution: To address these challenges, we propose MR-CLIP, a multimodal contrastive learning framework that aligns MR images with their DICOM metadata to learn contrast-aware representations, without manual labels. Trained on diverse clinical data spanning various scanners and protocols, MR-CLIP captures contrast variation across acquisitions and within scans, enabling anatomy-independent representation learning.
MR-CLIP/
├── preprocessing.py # Main preprocessing script
├── preprocessing.ipynb # Jupyter notebook version of preprocessing
├── bin_intervals_et_20_rt_20.json # Binning intervals for Echo Time and Repetition Time
├── requirements.txt # Python dependencies
└── src/ # Source code for training and evaluation
# Create and activate conda environment
conda create -n mr-clip python=3.8 -y
conda activate mr-clip
# Install dependencies
pip install -r requirements.txt
The preprocessing pipeline consists of four main steps:
-
NIfTI to PNG Conversion
- Converts NIfTI files to PNG images
- Determines scanning plane (axial, coronal, sagittal)
- Normalizes pixel values to 0-255
- Saves only nonzero slices
-
CSV Creation
- Creates CSV files with image paths and metadata
- Extracts and simplifies DICOM metadata
- Organizes data in batches for efficient processing
-
Data Labeling
- Creates unique labels for parameter combinations
- Bins numerical values (Echo Time, Repetition Time)
- Tracks label distributions
-
Data Splitting
- Merges all labeled data
- Shuffles while keeping slices together
- Splits into train/val/test sets
To run the preprocessing:
# Open and run the preprocessing notebook
jupyter notebook preprocessing.ipynb-
Download Pre-trained Weights
- Download the pre-trained weights from [20x20 Weights]
- Place the weights file in the
logs/mr_clip/checkpointsdirectory
-
Prepare Your Test Data
- Ensure your test data is in CSV format with columns:
filepath: Path to the PNG imagetext: Text description/metadatalabel: Numerical label (if available)
- Ensure your test data is in CSV format with columns:
-
Run Testing
cd src
python -m open_clip_train.main \
--report-to tensorboard \
--csv-separator=, \
--csv-img-key filepath \
--csv-caption-key text \
--val-data=/path/to/your/test_data.csv \
--batch-size=1000 \
--workers=8 \
--logs=/path/to/logs \
--device=cuda \
--dataset-type=csv \
--model=ViT-B-16 \
--name=mr_clip \
--resume=latest \
--distance \
--test \
--tracepredsKey parameters to adjust:
--val-data: Path to your test data CSV--batch-size: Adjust based on your GPU memory--workers: Number of data loading workers--logs: Directory for saving logs--name: Name of the experiment under logs folder. Place the weights under this folder.
With the --tracepreds option, you can check the results of retrieved predictions case by case which are saved under mr_clip/checkpoints .
If you use this code in your research, please cite:
@misc{avci2025mrclipefficientmetadataguidedlearning,
title={MR-CLIP: Efficient Metadata-Guided Learning of MRI Contrast Representations},
author={Mehmet Yigit Avci and Pedro Borges and Paul Wright and Mehmet Yigitsoy and Sebastien Ourselin and Jorge Cardoso},
year={2025},
eprint={2507.00043},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2507.00043},
}For questions and issues, please open an issue in this repository.
Our code repository is mainly built on OpenCLIP. We thank the authors for releasing their code.