NeuroTechX · bruAristimunha · Aug 1, 2023 · Jun 22, 2023 · Jun 23, 2023 · Jun 24, 2023
diff --git a/docs/source/datasets.rst b/docs/source/datasets.rst
@@ -95,6 +95,7 @@ Base & Utils
     :template: class.rst
 
     base.BaseDataset
+    base.CacheConfig
     fake.FakeDataset
     fake.FakeVirtualRealityDataset
 

diff --git a/docs/source/paradigms.rst b/docs/source/paradigms.rst
@@ -28,8 +28,8 @@ P300 Paradigms
     :toctree: generated/
     :template: class.rst
 
-	 SinglePass
-	 P300
+     SinglePass
+     P300
 
 ---------------
 SSVEP Paradigms
@@ -39,8 +39,19 @@ SSVEP Paradigms
     :toctree: generated/
     :template: class.rst
 
-	 SSVEP
-	 FilterBankSSVEP
+    SSVEP
+    FilterBankSSVEP
+
+--------------
+Fixed Interval Windows Processings
+--------------
+
+.. autosummary::
+    :toctree: generated/
+    :template: class.rst
+
+    FixedIntervalWindowsProcessing
+    FilterBankFixedIntervalWindowsProcessing
 
 ------------
 Base & Utils
@@ -53,6 +64,8 @@ Base & Utils
     motor_imagery.BaseMotorImagery
     motor_imagery.SinglePass
     motor_imagery.FilterBank
-	 p300.BaseP300
-	 ssvep.BaseSSVEP
+    p300.BaseP300
+    ssvep.BaseSSVEP
+    BaseFixedIntervalWindowsProcessing
     base.BaseParadigm
+    base.BaseProcessing
diff --git a/docs/source/whats_new.rst b/docs/source/whats_new.rst
@@ -29,25 +29,36 @@ Enhancements
 - Adding example to load different type of models (:gh:`401` by `Bruno Aristimunha`_ and `Igor Carrara`_)
 - Add resting state paradigm with dataset and example (:gh:`400` by `Gregoire Cattan`_ and `Pedro L. C. Rodrigues`_)
 - Speeding the augmentation method by 400% with NumPy vectorization  (:gh:`419` by `Bruno Aristimunha`_)
-- Improving the review processing with more pre-commit bots (:gh:`434` by `Bruno Aristimunha`_)
+- Add possibility to convert datasets to BIDS, plus `example <examples/example_bids_conversion.py>`_ (PR :gh:`408`, PR :gh:`391` by `Pierre Guetschel`_ and `Bruno Aristimunha`_)
+- Allow caching intermediate processing steps on disk, plus `example <examples/example_disk_cache.py>`_ (PR :gh:`408`, issue :gh:`385` by `Pierre Guetschel`_)
+- Restructure the paradigms and datasets to move all preprocessing steps to :mod:`moabb.datasets.preprocessing` and as sklearn pipelines (PR :gh:`408` by `Pierre Guetschel`_)
+- Add :func:`moabb.paradigms.FixedIntervalWindowsProcessing` and :func:`moabb.paradigms.FilterBankFixedIntervalWindowsProcessing`, plus `example <examples/example_fixed_interval_windows.py>`_ (PR :gh:`408`, issue :gh:`424` by `Pierre Guetschel`_)
+- Define :func:`moabb.paradigms.base.BaseProcessing`, common parent to :func:`moabb.paradigms.base.BaseParadigm` and :func:`moabb.paradigms.BaseFixedIntervalWindowsProcessing` (PR :gh:`408` by `Pierre Guetschel`_)
+- Allow passing a fixed processing pipeline to :func:`moabb.paradigms.base.BaseProcessing.get_data` and cache its result on disk (PR :gh:`408`, issue :gh:`367` by `Pierre Guetschel`_)
+- Update :func:`moabb.datasets.fake.FakeDataset`'s code to be unique for each parameter combination (PR :gh:`408` by `Pierre Guetschel`_)
+- Systematically set the annotations when loading data, eventually using the stim channel (PR :gh:`408` by `Pierre Guetschel`_)
+- Allow :func:`moabb.datasets.utils.dataset_search` to search across paradigms ``paradigm=None`` (PR :gh:`408` by `Pierre Guetschel`_)
+- Improving the review processing with more pre-commit bots (:gh:`435` by `Bruno Aristimunha`_)
 
 Bugs
 ~~~~
 
 - Restore 3 subject from Cho2017 (:gh:`392` by `Igor Carrara`_ and `Sylvain Chevallier`_)
 - Correct downloading with VirtualReality BrainInvaders dataset (:gh:`393` by `Gregoire Cattan`_)
-- Rename event `substraction` to `subtraction` in :func:`moabb.dataset.Shin2017B` (:gh:`397` by `Pierre Guetschel`_)
-- Save parameters of :func:`moabb.dataset.PhysionetMI` (:gh:`403` by `Pierre Guetschel`_)
+- Rename event `substraction` to `subtraction` in :func:`moabb.datasets.Shin2017B` (:gh:`397` by `Pierre Guetschel`_)
+- Save parameters of :func:`moabb.datasets.PhysionetMI` (:gh:`403` by `Pierre Guetschel`_)
 - Fixing issue with parallel evaluation (:gh:`401` by `Bruno Aristimunha`_ and `Igor Carrara`_)
 - Fixing SSLError from BCI competition IV (:gh:`404` by `Bruno Aristimunha`_)
-- Fixing :func:`moabb.dataset.bnci.MNEBNCI.data_path` that returned the data itself instead of paths (:gh:`412` by `Pierre Guetschel`_)
+- Fixing :func:`moabb.datasets.bnci.MNEBNCI.data_path` that returned the data itself instead of paths (:gh:`412` by `Pierre Guetschel`_)
 - Adding :func:`moabb.datasets.fake` in the init file to use in braindecode object (:gh:`414` by `Bruno Aristimunha`_)
 - Fixing the parallel download issue when the dataset have the same directory (:gh:`421` by `Sara Sedlar`_)
 - Fixing fixes the problem with the annotation loading for the P300 datasets Sosulski2019, Huebner2017 and Huebner2018 (:gh:`396` by `Sara Sedlar`_)
 - Removing the print in the dataset list (:gh:`423` by `Bruno Aristimunha`_)
 - Fixing bug in :func:`moabb.pipeline.utils_pytorch.BraindecodeDatasetLoader` where incorrect y was used in transform calls (:gh:`426` by `Gabriel Schwartz`_)
 - Fixing one test in :func:`moabb.pipeline.utils_pytorch.BraindecodeDatasetLoader` (:gh:`426` by `Bruno Aristimunha`_)
-
+- Fix :func:`moabb.benchmark` overwriting ``include_datasets`` list (:gh:`408` by `Pierre Guetschel`_)
+- Fix :func:`moabb.paradigms.base.BaseParadigm` using attributes before defining them  (PR :gh:`408`, issue :gh:`425` by `Pierre Guetschel`_)
+- Fix :func:`moabb.paradigms.FakeImageryParadigm`, :func:`moabb.paradigms.FakeP300Paradigm` and :func:`moabb.paradigms.FakeSSVEPParadigm` ``is_valid`` methods to only accept the correct datasets (PR :gh:`408` by `Pierre Guetschel`_)
 
 API changes
 ~~~~~~~~~~~

diff --git a/examples/plot_bids_conversion.py b/examples/plot_bids_conversion.py
@@ -0,0 +1,133 @@
+"""
+===============================
+Convert a MOABB dataset to BIDS
+===============================
+
+The Brain Imaging Data Structure (BIDS) format
+is standard for storing neuroimaging data.
+It follows fixed principles to facilitate the
+sharing of neuroimaging data between researchers.
+
+The MOABB library allows to convert any MOABB dataset to
+BIDS [1]_ and [2]_.
+
+In this example, we will convert the AlexMI dataset to BIDS using the
+option `cache_config=dict(path=temp_dir, save_raw=True)` of the get_data
+method from the dataset object.
+
+This will automatically save the raw data in the BIDS format and allow to use
+a cache for the next time the dataset is used.
+
+We will use the AlexMI dataset [3]_, one of the smallest in
+people and one that can be downloaded quickly.
+
+References
+-----------
+
+.. [1] Pernet, C.R., Appelhoff, S., Gorgolewski, K.J. et al. EEG-BIDS,
+       An extension to the brain imaging data structure for
+       electroencephalography. Sci Data 6, 103 (2019).
+       https://doi.org/10.1038/s41597-019-0104-8
+
+.. [2] Appelhoff et al., (2019). MNE-BIDS: Organizing electrophysiological
+       data into the BIDS format and facilitating their analysis.
+       Journal of Open Source Software, 4(44), 1896,
+       https://doi.org/10.21105/joss.01896
+
+.. [3] Barachant, A., 2012. Commande robuste d'un effecteur par une
+       interface cerveau machine EEG asynchrone (Doctoral dissertation,
+       Université de Grenoble).
+       https://tel.archives-ouvertes.fr/tel-01196752
+
+"""
+# Authors: Pierre Guetschel <[email protected]>
+#
+# License: BSD (3-clause)
+
+import shutil
+import tempfile
+from pathlib import Path
+
+import mne
+
+from moabb import set_log_level
+from moabb.datasets import AlexMI
+
+
+set_log_level("info")
+
+###############################################################################
+# Basic usage
+# -----------
+#
+# Here, we will save the BIDS version of the dataset in a temporary folder
+temp_dir = Path(tempfile.mkdtemp())
+# The conversion of any MOABB dataset to a BIDS-compliant structure can be done
+# by simply calling its ``get_data`` method and using the ``cache_config``
+# parameter. This parameter is a dictionary.
+dataset = AlexMI()
+_ = dataset.get_data(cache_config=dict(path=temp_dir, save_raw=True))
+
+
+###############################################################################
+# Before / after folder structure
+# -----------------------------
+#
+# To investigate what was saved, we will first define a function to print
+# the folder structure of a given path:
+def print_tree(p: Path, last=True, header=""):
+    elbow = "└──"
+    pipe = "│  "
+    tee = "├──"
+    blank = "   "
+    print(header + (elbow if last else tee) + p.name)
+    if p.is_dir():
+        children = list(p.iterdir())
+        for i, c in enumerate(children):
+            print_tree(
+                c, header=header + (blank if last else pipe), last=i == len(children) - 1
+            )
+
+
+###############################################################################
+# Now, we will retrieve the location of the original dataset. It is stored
+# in the MNE data directory, which can be found with the ``"MNE_DATA"`` key:
+mne_data = Path(mne.get_config("MNE_DATA"))
+print(f"MNE data directory: {mne_data}")
+
+###############################################################################
+# Now, we can print the folder structure of the original dataset:
+print("Before conversion:")
+print_tree(mne_data / "MNE-alexeeg-data")
+
+###############################################################################
+# As we can see, before conversion, all the data (i.e. from all subjects,
+# sessions and runs) is stored in a single folder. This follows no particular
+# standard and can vary from one dataset to another.
+#
+# After conversion, the data is stored in a BIDS-compliant way:
+print("After conversion:")
+print_tree(temp_dir / "MNE-alexandre motor imagery-bids-cache")
+
+###############################################################################
+# In the BIDS version of our dataset, the raw files are saved in EDF.
+# The data is organized in a hierarchy of folders,
+# starting with the subjects, then the sessions, and then the runs. Metadata
+# files are stored to describe the data. For more details on the BIDS
+# structure, please refer to the `BIDS website <https://bids.neuroimaging.io>`_
+# and the `BIDS spec <https://bids-specification.readthedocs.io/en/stable/>`_.
+#
+# Under the hood, saving datasets to BIDS is done through the caching system
+# of MOABB. Only raw EEG files are officially supported by the BIDS
+# specification.
+# However, MOABB's caching mechanism also offers the possibility to save
+# the data in a pseudo-BIDS after different preprocessing steps.
+# In particular, we can save :class:`mne.Epochs` and ``np.ndarray`` objects.
+# For more details on the caching system,
+# please refer to the tutorial :doc:`./plot_disk_cache`.
+#
+# Cleanup
+# -------
+#
+# Finally, we can delete the temporary folder:
+shutil.rmtree(temp_dir)