[MRG][FIX] Fixes to examples #2990

jaeilepp · 2016-03-07T08:37:30Z

Noticed that with the new SPM datasets the examples are not working. See plot_spm_faces_datasets for example.

agramfort · 2016-03-07T09:31:30Z

hum... what's wrong?

jaeilepp · 2016-03-07T10:56:24Z

Yeah. I can also look deeper later, but I was asking if anyone (@Eric89GXL ) has an idea...

agramfort · 2016-03-07T11:01:56Z

compare for now the entries in the meas info. We have utility function for this.

larsoner · 2016-03-07T14:29:08Z

I'm not sure what the difference would be. You could try converting the files using the C tools and swap in the .fif files for the CTF ones and see if you get the same or different results.

jaeilepp · 2016-03-09T14:05:44Z

Seems like with our python reader the data has some weird stuff at the start causing the events to be out of sync. This mess is not appearing with the converted fif file:

larsoner · 2016-03-09T14:11:08Z

That looks like a bug. I wonder why it's not caught by the tests. Can you see if the verbose output is similar to what the C code says? CTF data has some skips, and they might not match properly. I probably won't have time to check for a while.

jaeilepp · 2016-03-09T14:36:09Z

Seems that the mess at the beginning is data from the end of the recording. I see similar patterns when comparing to the end of the c converted file. You can also see the reference signal on the first channel:

jaeilepp · 2016-03-10T10:44:55Z

mne/io/ctf/ctf.py

@@ -149,13 +150,13 @@ def _read_segment_file(self, data, idx, fi, start, stop, cals, mult):
        with open(self._filenames[fi], 'rb') as fid:
            for bi in range(len(r_lims)):
                samp_offset = (bi + trial_start_idx) * si['res4_nsamp']
-                n_read = min(si['n_samp'] - samp_offset, si['block_size'])
+                n_read = min(si['n_samp_tot'] - samp_offset, si['block_size'])


This was the bug. It wasn't caught by the tests because on our testing data n_samp and n_samp_tot are equal.

no need to update a test?

As I said, our testing data has equal values for these. We should probably add dataset with different values. I wonder if @dengemann or someone else has nice small files.

And I'm not entirely sure this is the correct solution to fix the bug. It almost looks like our python version tries to leave out some samples from the end (see https://github.com/mne-tools/mne-python/blob/master/mne/io/ctf/ctf.py#L214), but at least with our spm_faces dataset this wasn't behaving well. Maybe someone with deeper understanding of CTF format can help.

Both the Python and C versions can omit samples from the end. We do it differently from the C tools to handle a user's use case, see:

#2712

I suspect this fix will break that use case, but I'm not sure. We really do need to add some test files to figure it out.

jaeilepp · 2016-03-10T13:31:45Z

If we are content with the ctf fix, this is ready for review/merge. It can be fixed in another PR if needed.

agramfort · 2016-03-10T20:58:42Z

I would check against brainstorm reader to make sure we read the correct number of samples and proper values of everything...

larsoner · 2016-03-11T02:44:23Z

You'll need a rebase since I decreased mem usage in the SPM example

jaeilepp · 2016-03-11T08:30:29Z

Come to think of it, we should probably make it throw an error if baseline is not None and mode is None to avoid people thinking they're baseline correcting when they're not. And those checks should be done in rescale.

Now this happens. It might break someone's code, but that means they've been doing it wrong all along.

To me it looks like the ctf bug fix here is the right one. Calling with system_clock='ignore' on master makes the artefacts in the beginning disappear, but it also added samples to the end, which doesn't happen with c version. With this fix, 'ignore' adds those samples too, but the artefacts aren't there either way. So judging by the behavior, the fix works.

larsoner · 2016-03-11T12:43:33Z

Yeah the C code doesn't have an option to ignore to clock IIRC. So now the raw data looks correct, the example renders properly, and tests pass? Can you add a test that requires the spm face dataset to catch the regression? It won't run on Travis but it will on most dev machines.

jaeilepp · 2016-03-11T14:21:43Z

Okay test added. Now it just checks that samples are omitted. For data comparison, the data should be added to the dataset. I believe this is sufficient for now.
I also removed special characters from maxwell.py. They were causing failures with sphinx-gallery.
Ready for review from my end.

larsoner · 2016-03-11T14:25:19Z

Can we improve sphinx gallery instead of removing those characters? There are multiple other files that have them as well, and they are convenient to have. Probably better to open an issue to see if we can deal with it properly instead of tacking it on here.

agramfort · 2016-03-11T14:30:14Z

examples/visualization/plot_clickable_image.py


 # We've already clicked and exported
-layout_path = op.join(op.dirname(mne.__file__), 'data', 'image')
+layout_path = op.join(op.dirname(__file__), '..', '..', 'mne', 'data', 'image')


you cannot assume the example file is at a fixed place on disk. The previous option is more robust.

It fails on circle. That's why I changed it.

What's the failure? Is there some way to use the location of mne instead of the location of the example file? It really should be more robust to use the repo-location to find data files from the repo than use the example-location -- people could reasonably just download the example to their Desktop and expect it to work. Maybe we need to add mne.data_path within the repo that just points to op.join(os.abspath(__file__), 'data') or so in __init__.py.

See https://circle-artifacts.com/gh/mne-tools/mne-python/451/artifacts/1/home/ubuntu/mne-python/doc/_build/html/auto_examples/visualization/plot_clickable_image.html#sphx-glr-auto-examples-visualization-plot-clickable-image-py

I suspect that the gif file is not properly installed via our setup.py .... it's a bug. The example should not be changed due to this.

Good call, it is indeed missing

https://github.com/mne-tools/mne-python/blob/master/setup.py#L106

agramfort · 2016-03-11T14:33:02Z

I agree with @Eric89GXL.

the bug should be fixed in sphinx-gallery.

Please open an issue there and cc me.

thx

jaeilepp · 2016-03-11T15:00:03Z

To me allowing special characters sounds like asking for trouble. The doc builds are way too fragile to some specific versions as it is.

agramfort · 2016-03-11T15:02:48Z

at least raise the issue on sphinx gallery. They are really responsive.

larsoner · 2016-03-11T15:33:40Z

UTF8 is pretty standard (and default on Py3k) it should really be supported. I'd rather not mangle our docstrings if we don't have to

larsoner · 2016-03-11T16:44:55Z

mne/baseline.py

@@ -55,15 +59,12 @@ def rescale(data, times, baseline, mode='mean', copy=True, verbose=None):
    data_scaled: array
        Array of same shape as data after rescaling.
    """
+    _log_rescale(baseline, mode)
+    if baseline is None:
+        return data


This changes the previous reasonable behavior of the function -- before if copy is True then it would always return a copy, now it does in some cases and doesn't in others.

larsoner · 2016-03-12T14:17:43Z

Awesome, thanks

agramfort · 2016-03-13T17:54:54Z

let me know if we're good here.

larsoner · 2016-03-14T01:48:06Z

This should fix sphinx-gallery:

sphinx-gallery/sphinx-gallery#106

jaeilepp · 2016-03-14T09:39:17Z

Ready for review.

larsoner · 2016-03-14T14:09:53Z

examples/inverse/plot_covariance_whitening_dspm.py


-raw = io.Raw(raw_fname % 1)  # Take first run
+raw = io.read_raw_ctf(raw_fname % 1, preload=True)  # Take first run


You shouldn't need preload=True, if you do, we have a bug

(this will increase memory usage substantially)

Whoops. I don't know how that got there. But in any case there is a load_data on the line below so this seems redundant.

More importantly, the load_data happens after the crop statement, which leads to memory savings

agramfort · 2016-03-15T17:18:55Z

besides LGTM

don't forget to update what's new BUG section

…orrection.

jaeilepp · 2016-03-16T09:03:12Z

I thought I had found a setting that would work with mayavi figures, but apparently I was wrong. I'll try it on another PR then.
Ready for review.

agramfort · 2016-03-16T09:10:51Z

Great

Merge when Travis is happy

[MRG][FIX] Fixes to examples

jaeilepp force-pushed the fix-examples branch from 658d883 to cfc5be6 Compare March 10, 2016 10:43

jaeilepp reviewed Mar 10, 2016
View reviewed changes

jaeilepp changed the title ~~[WIP][FIX] Fixes to examples~~ [MRG][FIX] Fixes to examples Mar 10, 2016

jaeilepp force-pushed the fix-examples branch from 24daf86 to 0f01f70 Compare March 11, 2016 08:09

agramfort reviewed Mar 11, 2016
View reviewed changes

larsoner reviewed Mar 11, 2016
View reviewed changes

jaeilepp force-pushed the fix-examples branch from 1aad3fd to f9774a3 Compare March 14, 2016 07:22

larsoner reviewed Mar 14, 2016
View reviewed changes

jaeilepp added 15 commits March 16, 2016 09:55

Changed spm datapath.

31ee5b3

Fixed a bug in ctf reader. Correct path for eegbci. Fix to baseline c…

fc3868e

…orrection.

raw_path -> raw_fname

b5f3560

Fix to image path.

57155b4

Removed baseline from example.

daf3bdf

Fix to baseline.

4f9c602

sphinx version.

21a5f60

Added test. Removed special characters.

d806c31

Return special characters. Fixes.

d7917bf

Fix to path.

5a1655c

Cleaning.

096b184

Added pandas.

b5fda5b

Fix to setup.py.

89596f9

Fix

b412271

Fix to data_path. Updated whats_new.

0be5d45

jaeilepp force-pushed the fix-examples branch from 45a99ce to 0be5d45 Compare March 16, 2016 09:01

jaeilepp added a commit that referenced this pull request Mar 16, 2016

Merge pull request #2990 from jaeilepp/fix-examples

3e79f10

[MRG][FIX] Fixes to examples

jaeilepp merged commit 3e79f10 into mne-tools:master Mar 16, 2016

jaeilepp deleted the fix-examples branch March 16, 2016 09:39


		raw = io.Raw(raw_fname % 1) # Take first run
		raw = io.read_raw_ctf(raw_fname % 1, preload=True) # Take first run

[MRG][FIX] Fixes to examples #2990

[MRG][FIX] Fixes to examples #2990

Uh oh!

Conversation

jaeilepp commented Mar 7, 2016

Uh oh!

agramfort commented Mar 7, 2016

Uh oh!

jaeilepp commented Mar 7, 2016

Uh oh!

agramfort commented Mar 7, 2016 via email

Uh oh!

larsoner commented Mar 7, 2016

Uh oh!

jaeilepp commented Mar 9, 2016

Uh oh!

larsoner commented Mar 9, 2016

Uh oh!

jaeilepp commented Mar 9, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaeilepp commented Mar 10, 2016

Uh oh!

agramfort commented Mar 10, 2016

Uh oh!

larsoner commented Mar 11, 2016

Uh oh!

jaeilepp commented Mar 11, 2016

Uh oh!

larsoner commented Mar 11, 2016 via email

Uh oh!

jaeilepp commented Mar 11, 2016

Uh oh!

larsoner commented Mar 11, 2016 via email

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agramfort commented Mar 11, 2016

Uh oh!

jaeilepp commented Mar 11, 2016

Uh oh!

agramfort commented Mar 11, 2016 via email

Uh oh!

larsoner commented Mar 11, 2016 via email

Uh oh!

Choose a reason for hiding this comment

Uh oh!

larsoner commented Mar 12, 2016 via email

Uh oh!

agramfort commented Mar 13, 2016

Uh oh!

larsoner commented Mar 14, 2016

Uh oh!

jaeilepp commented Mar 14, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!