Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

drammock
Copy link
Member

@drammock drammock commented Mar 25, 2025

The export code for EDF files was calling np.pad incorrectly, causing the channels axis to also get padded with however many extra samples were needed to make equal-sized data blocks along the time axis. Marking as draft because it's the end of the day and I haven't had time to write a test yet. This should shrink almost all exported EDF files, possibly quite significantly. EDIT: the error is self-correcting in the output, because the extra channels are never added to the exported file (there's a for-loop over the channels in the original raw). But it is a bug in that it triggers allocating an unnecessarily large intermediate array, and for me was leading to MemoryErrors when the pad value was big.

https://numpy.org/doc/stable/reference/generated/numpy.pad.html

@drammock
Copy link
Member Author

This turns out to be tricky to test, because the array that is created with the padded values doesn't survive the exporting process. But here's a demonstration:

(Pdb) p pad_width
999
(Pdb) p data.shape
(4, 1001)
(Pdb) data2 = np.pad(data, (0, int(pad_width)), "edge")
(Pdb) p data2.shape
(1003, 2000)  # <----- went from 4 channels to 1003 channels

the reason it doesn't show up in the output is that there's later a loop over the channels in the original raw so the excess spurious channels (which are always at the end) are never added to the list of EDFSignals that ultimately get written to file. In lieu of a test, I've stuck a couple assert statements inline that cause the existing test_edf_padding test to fail on main.

@drammock drammock marked this pull request as ready for review March 26, 2025 22:32
Copy link
Contributor

@cbrnr cbrnr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I didn't know that padding was applied to all axes either, but it makes sense.

@drammock
Copy link
Member Author

drammock commented Mar 27, 2025

CI fails look like they might all be due to random number generator issues?

Linux pip-pre

  • assert_allclose: mne/forward/tests/test_make_forward.py::test_make_forward_dipole:721
  • assert_allclose: mne/preprocessing/tests/test_fine_cal.py::test_fine_cal_systems[fil]:292
  • mne/preprocessing/tests/test_maxwell.py::test_other_systems - RuntimeWarning: Estimated head radius (10.9 cm) is above the 99th percentile for adult head size
  • assert median difference < tolerance: mne/simulation/tests/test_raw.py::test_simulate_raw_bem[testing_data]:398

Windows pip-pre

  • mne/preprocessing/tests/test_fine_cal.py::test_compute_fine_cal[VectorView] - assert 62 < np.float64(61.578209335821576)
  • mne/preprocessing/tests/test_fine_cal.py::test_fine_cal_systems[kit] - AssertionError: Shielding factor not 2.300 <= 2.275 < 2.800
    assert 2.3 <= np.float64(2.2748473564245675)
  • mne/preprocessing/tests/test_fine_cal.py::test_fine_cal_systems[ctf] - RuntimeError: No usable segments found
  • mne/preprocessing/tests/test_fine_cal.py::test_fine_cal_systems[fil] - AssertionError: Shielding factor not 125.000 <= 152.489 < 145.000
    assert np.float64(152.4889126829783) < 145
  • mne/decoding/tests/test_search_light.py::test_search_light_basic

Locally I have NumPy 2.1.3 (currently numba forbids anything higher), but regardless I can't find anything in the NumPy changelogs from 2.1.3 onward that seems likely to affect assert_allclose or NumPy's random generators (this is the closest: numpy/numpy#28006). Needs further investigation.

@larsoner
Copy link
Member

Often these things crop up due to some change in linalg, like an update to OpenBLAS for example. It changes slightly the results of pseudo-inversion etc. and so some tolerances are too strict. We usually just bump them as long as they aren't way out of range. These all look bumpable to me

@drammock
Copy link
Member Author

Often these things crop up due to some change in linalg, like an update to OpenBLAS for example. It changes slightly the results of pseudo-inversion etc. and so some tolerances are too strict. We usually just bump them as long as they aren't way out of range. These all look bumpable to me

OK, I bumped whatever I could figure out how to bump. I suspect that these two won't actually be fixed by my changes though:

  • mne/preprocessing/tests/test_maxwell.py::test_other_systems - RuntimeWarning: Estimated head radius (10.9 cm) is above the 99th percentile for adult head size (because I was unable to reproduce, I wasn't even sure which system was causing the warning)

  • mne/decoding/tests/test_search_light.py::test_search_light_basic (unexpected skip: sklearn int_t / long long mismatch which AFAICT should not be happening anymore?)

@larsoner
Copy link
Member

Okay I think the differences are due to changes in fmin_cobyla actually. In theory the results should now be more accurate, but it does mean that some of our checks will need to change a bit

@larsoner
Copy link
Member

... I'll merge this one as-is and then open a separate PR for the cobyla stuff.

@larsoner larsoner merged commit e04c4d9 into mne-tools:main Mar 31, 2025
27 of 30 checks passed
@larsoner
Copy link
Member

Thanks @drammock !

@drammock drammock deleted the fix-edf-export branch March 31, 2025 17:40
larsoner added a commit to larsoner/mne-python that referenced this pull request Apr 4, 2025
* upstream/main: (149 commits)
  FIX make_watershed_bem to handle missing talairach_with_skull.lta courtesy Freesurfer 8 (mne-tools#13172)
  ENH: Add upsampling for MEG helmet surface (mne-tools#13179)
  MAINT: Update code credit (mne-tools#13180)
  BUG: Fix bug with least-squares sphere fit (mne-tools#13178)
  fix EDF export (mne-tools#13174)
  fix typo (mne-tools#13171)
  [pre-commit.ci] pre-commit autoupdate (mne-tools#13164)
  Fix dev installation guide (mne-tools#13163)
  expose 'mode' for plotting dipole on brain (mne-tools#13162)
  turn dipole attrs into properties (mne-tools#13153)
  remove misformatted (and unused) crossref anchor (mne-tools#13155)
  doc: point to read_dipole (mne-tools#13149)
  [pre-commit.ci] pre-commit autoupdate (mne-tools#13152)
  BUG: Fix bug with not short-circuiting n_jobs=1 (mne-tools#13147)
  FIX: Missing coordinates.xml in MFF file (mne-tools#13148)
  FIX: Gracefully handle bad XML files in EGI reader (mne-tools#13145)
  Fixes for Latest IPython (9.0.1) (mne-tools#13146)
  Fix intersphinx (mne-tools#13143)
  BUG: Fix bug with parallel doc build (mne-tools#13140)
  [pre-commit.ci] pre-commit autoupdate (mne-tools#13141)
  ...
larsoner added a commit to SYXiao2002/mne-python that referenced this pull request Apr 18, 2025
* upstream/main: (40 commits)
  fix typo (missing space) that messed up rst rendering (mne-tools#13217)
  MAINT: Restore VTK dev (mne-tools#13214)
  [pre-commit.ci] pre-commit autoupdate (mne-tools#13212)
  BUG: Fix bug with example (mne-tools#13210)
  MAINT: Fix pip-pre with PyVista (mne-tools#13207)
  Move FCBG to former partners (mne-tools#13205)
  ENH: Update related software list (mne-tools#13202)
  fix sfreq estimation for snirf files (mne-tools#13184)
  ENH: Use data-based padding instead of "odd" padding when filtering in raw.plot (mne-tools#13183)
  FIX: Bumps (mne-tools#13198)
  DOC: fix typo in examples/io/read_impedances.py (mne-tools#13197)
  [pre-commit.ci] pre-commit autoupdate (mne-tools#13173)
  FIX make_watershed_bem to handle missing talairach_with_skull.lta courtesy Freesurfer 8 (mne-tools#13172)
  ENH: Add upsampling for MEG helmet surface (mne-tools#13179)
  MAINT: Update code credit (mne-tools#13180)
  BUG: Fix bug with least-squares sphere fit (mne-tools#13178)
  fix EDF export (mne-tools#13174)
  fix typo (mne-tools#13171)
  [pre-commit.ci] pre-commit autoupdate (mne-tools#13164)
  Fix dev installation guide (mne-tools#13163)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants