-
-
Notifications
You must be signed in to change notification settings - Fork 1.4k
fix EDF export #13174
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix EDF export #13174
Conversation
This turns out to be tricky to test, because the array that is created with the padded values doesn't survive the exporting process. But here's a demonstration:
the reason it doesn't show up in the output is that there's later a loop over the channels in the original raw so the excess spurious channels (which are always at the end) are never added to the list of |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch! I didn't know that padding was applied to all axes either, but it makes sense.
CI fails look like they might all be due to random number generator issues? Linux pip-pre
Windows pip-pre
Locally I have NumPy 2.1.3 (currently numba forbids anything higher), but regardless I can't find anything in the NumPy changelogs from 2.1.3 onward that seems likely to affect |
Often these things crop up due to some change in linalg, like an update to OpenBLAS for example. It changes slightly the results of pseudo-inversion etc. and so some tolerances are too strict. We usually just bump them as long as they aren't way out of range. These all look bumpable to me |
OK, I bumped whatever I could figure out how to bump. I suspect that these two won't actually be fixed by my changes though:
|
Okay I think the differences are due to changes in |
... I'll merge this one as-is and then open a separate PR for the cobyla stuff. |
Thanks @drammock ! |
* upstream/main: (149 commits) FIX make_watershed_bem to handle missing talairach_with_skull.lta courtesy Freesurfer 8 (mne-tools#13172) ENH: Add upsampling for MEG helmet surface (mne-tools#13179) MAINT: Update code credit (mne-tools#13180) BUG: Fix bug with least-squares sphere fit (mne-tools#13178) fix EDF export (mne-tools#13174) fix typo (mne-tools#13171) [pre-commit.ci] pre-commit autoupdate (mne-tools#13164) Fix dev installation guide (mne-tools#13163) expose 'mode' for plotting dipole on brain (mne-tools#13162) turn dipole attrs into properties (mne-tools#13153) remove misformatted (and unused) crossref anchor (mne-tools#13155) doc: point to read_dipole (mne-tools#13149) [pre-commit.ci] pre-commit autoupdate (mne-tools#13152) BUG: Fix bug with not short-circuiting n_jobs=1 (mne-tools#13147) FIX: Missing coordinates.xml in MFF file (mne-tools#13148) FIX: Gracefully handle bad XML files in EGI reader (mne-tools#13145) Fixes for Latest IPython (9.0.1) (mne-tools#13146) Fix intersphinx (mne-tools#13143) BUG: Fix bug with parallel doc build (mne-tools#13140) [pre-commit.ci] pre-commit autoupdate (mne-tools#13141) ...
* upstream/main: (40 commits) fix typo (missing space) that messed up rst rendering (mne-tools#13217) MAINT: Restore VTK dev (mne-tools#13214) [pre-commit.ci] pre-commit autoupdate (mne-tools#13212) BUG: Fix bug with example (mne-tools#13210) MAINT: Fix pip-pre with PyVista (mne-tools#13207) Move FCBG to former partners (mne-tools#13205) ENH: Update related software list (mne-tools#13202) fix sfreq estimation for snirf files (mne-tools#13184) ENH: Use data-based padding instead of "odd" padding when filtering in raw.plot (mne-tools#13183) FIX: Bumps (mne-tools#13198) DOC: fix typo in examples/io/read_impedances.py (mne-tools#13197) [pre-commit.ci] pre-commit autoupdate (mne-tools#13173) FIX make_watershed_bem to handle missing talairach_with_skull.lta courtesy Freesurfer 8 (mne-tools#13172) ENH: Add upsampling for MEG helmet surface (mne-tools#13179) MAINT: Update code credit (mne-tools#13180) BUG: Fix bug with least-squares sphere fit (mne-tools#13178) fix EDF export (mne-tools#13174) fix typo (mne-tools#13171) [pre-commit.ci] pre-commit autoupdate (mne-tools#13164) Fix dev installation guide (mne-tools#13163) ...
The export code for EDF files was calling
np.pad
incorrectly, causing the channels axis to also get padded with however many extra samples were needed to make equal-sized data blocks along the time axis. Marking as draft because it's the end of the day and I haven't had time to write a test yet.This should shrink almost all exported EDF files, possibly quite significantly.EDIT: the error is self-correcting in the output, because the extra channels are never added to the exported file (there's a for-loop over the channels in the original raw). But it is a bug in that it triggers allocating an unnecessarily large intermediate array, and for me was leading toMemoryError
s when the pad value was big.https://numpy.org/doc/stable/reference/generated/numpy.pad.html