ENH: Parallel gallery generation #877

jschueller · 2021-10-28T07:53:53Z

Closes #25

larsoner · 2021-10-28T14:59:40Z

Thinking about this a bit more, I'd expect this not to work for (at least) the matplotlib, mayavi, and pyvista scrapers because these are all global-state based. And then there will be tricky interactions with reset_modules, which also by default does global state stuff at least for matplotlib. So I'm not sure this will ever work at least for the majority of our users :(

larsoner · 2021-10-28T15:00:34Z

I'll copy that over to the issue so more people can tell me how I'm wrong :)

jschueller · 2022-01-11T09:13:49Z

maybe the sphinx folks (@samdoran or @tk0miya) could give hints, or sklearn (@Titan-C @GaelVaroquaux) would be interested in speeding up their builds too ?

tk0miya · 2022-01-11T17:12:05Z

It seems this handler is invoked on the bootstrap process of Sphinx (on builder-inited event). So there is no special support from Sphinx framework.

jschueller · 2022-01-11T17:25:45Z

indeed, we're looking at parallelizing the gallery jobs at the extension level

larsoner · 2024-06-04T22:27:14Z

@jschueller there has been some renewed interest in this so I'm taking a stab at actually making it work. I think it's close but I need to fix a test and maybe a couple of examples, we'll see!

jschueller · 2024-06-05T05:05:03Z

@jschueller there has been some renewed interest in this so I'm taking a stab at actually making it work. I think it's close but I need to fix a test and maybe a couple of examples, we'll see!

this is great!

jschueller

It seems to work! Thanks a lot
Do you plan a new release with this ?

larsoner · 2024-06-05T14:41:48Z

I still need to figure out how to test this. And also decide if joblib is the right thing to use for parallelization. I suspect if I just used multiprocessing with spawn then we maybe could keep memory measurements working, which would be nice. I have to think about it...

But yes I think once this is in and maybe @jschueller you and @lagru run some preliminary tests we could cut a release. I'll probably write in the docs that it's new/experimental, though, since I do expect there to be issues just because parallelization seems to always create them!

Actually test
Try multiprocessing maybe?
Add docs saying it's experimental

jschueller · 2024-06-05T14:52:49Z

I already tested it on a real project (openturns) and it just works

larsoner · 2024-06-24T14:53:13Z

Okay pushed a doc update -- @lucyleeow feel free to review and merge if happy when you get a chance!

larsoner · 2024-06-24T18:25:31Z

Okay I realized that for MNE-Python we treat all warnings in doc builds/examples as errors, so I needed to prevent raising the UserWarning that joblib emits as an error. Then I could test a few cases:

MNE-Python

No examples (noplot)

This build takes ~8 min.

All examples, parallel=1 (serial)

Four slowest examples took:

    - ../examples/datasets/opm_data.py:                                       57.88 sec   0.0 MB
    - ../examples/time_frequency/source_power_spectrum_opm.py:                55.67 sec   0.0 MB
    - ../examples/datasets/spm_faces_dataset.py:                              49.65 sec   0.0 MB
    - ../tutorials/io/60_ctf_bst_auditory.py:                                 45.00 sec   0.0 MB

the full doc build takes about an hour:

real	59m47.935s
user	54m54.940s

So ~52 minutes to build the examples.

All examples, parallel=4

My machine has 4 physical cores so I tried this next and saw a bit of CPU oversubscription by eye and looking at the four slowest examples:

    - ../examples/datasets/opm_data.py:                                       68.57 sec   0.0 MB
    - ../examples/inverse/multi_dipole_model.py:                              63.33 sec   0.0 MB
    - ../examples/datasets/spm_faces_dataset.py:                              57.83 sec   0.0 MB
    - ../tutorials/preprocessing/40_artifact_correction_ica.py:               50.90 sec   0.0 MB

but the overall build time was cut in half (hooray!):

real	27m9.492s
user	59m48.404s

And the example part of that is ~19min.

So our speedup for the example-running portion is ~52min down to ~19min! A factor of ~2.7 which is pretty good I think!

There is one fancy example that has a custom scraper in MNE that I don't think I could actually get to work properly in parallel based on what it does (creates HTML files, sets them to be copied later, etc.). I think we'll want something like a # sphinx_gallery_non_parallel comment or something that forces it to be run in non-parallel mode, but I can do that in a follow-up PR I think. I don't think it will be very complicated but don't want to add more to this PR!

lucyleeow · 2024-06-27T23:56:40Z

Sorry, I've missed the ping, I'll take a quick look today.

lucyleeow

Thanks! I pushed a small commit to note version support drop in CHANGES.rst

I know you've noted more fixes to do, happy to look again later.

lucyleeow · 2024-06-28T03:51:29Z

doc/conf.py

-    "sphinx": ("https://www.sphinx-doc.org/en/master", None),
-    "pandas": ("https://pandas.pydata.org/pandas-docs/stable/", None),
-}
+intersphinx_mapping = get_intersphinx_mapping(


sphinx_gallery/gen_gallery.py

lucyleeow · 2024-06-28T05:01:15Z

sphinx_gallery/gen_rst.py

+                    f"{gallery_conf['show_memory']=} disabled due to "
+                    f"{gallery_conf['parallel']=}."
+                )
+                gallery_conf["show_memory"] = False


Question - how is the update to gallery_conf["show_memory"] passed outside the function ?

As a dict, gallery_conf` is mutable so the change does not need to be passed back, changing it here changes it everywhere

Welp I returned gallery_conf unnecessarily in my configuration clean up PR then. Will fix this later

sphinx_gallery/gen_rst.py

lucyleeow · 2024-06-28T06:08:12Z

sphinx_gallery/gen_rst.py

+        elif "passing" in out_vars:
+            assert "stale" not in out_vars
+            gallery_conf["passing_examples"].append(src_file)
+        elif "stale" in out_vars:  # non-executable files have none of these three


what are "these three"?

Would it be clearer to say that 'stale' examples are 'not re-executed'? or something similar?

Changing to at the top of the three conditionals:

# n.b. non-executable files have none of these three variables defined, # so the last conditional must be "elif" not just "else"

sphinx_gallery/gen_rst.py

lucyleeow · 2024-06-28T06:21:09Z

sphinx_gallery/gen_rst.py

-    if sys.platform in ("win32", "darwin"):
-        sleep, timeout = (1, 2)
-    else:
-        sleep, timeout = (0.5, 1)
-    proc = subprocess.Popen(
-        [sys.executable, "-c", f"import time, sys; time.sleep({sleep}); sys.exit(0)"],
-        close_fds=True,
-    )
-    memories = memory_usage(proc, interval=1e-3, timeout=timeout)
-    proc.communicate(timeout=timeout)
-    # On OSX sometimes the last entry can be None
-    memories = [mem for mem in memories if mem is not None] + [0.0]
-    memory_base = max(memories)


Just for my info - why are we deleting? 😬

It never really worked robustly. On macOS it almost always errored to open and measure the subprocess

larsoner

Okay pushing a commit, let me know what you think @lucyleeow !

sphinx_gallery/gen_rst.py

larsoner · 2024-06-29T16:54:16Z

sphinx_gallery/gen_rst.py

+        elif "passing" in out_vars:
+            assert "stale" not in out_vars
+            gallery_conf["passing_examples"].append(src_file)
+        elif "stale" in out_vars:  # non-executable files have none of these three


Changing to at the top of the three conditionals:

# n.b. non-executable files have none of these three variables defined, # so the last conditional must be "elif" not just "else"

larsoner · 2024-06-29T16:54:54Z

sphinx_gallery/gen_rst.py

-    if sys.platform in ("win32", "darwin"):
-        sleep, timeout = (1, 2)
-    else:
-        sleep, timeout = (0.5, 1)
-    proc = subprocess.Popen(
-        [sys.executable, "-c", f"import time, sys; time.sleep({sleep}); sys.exit(0)"],
-        close_fds=True,
-    )
-    memories = memory_usage(proc, interval=1e-3, timeout=timeout)
-    proc.communicate(timeout=timeout)
-    # On OSX sometimes the last entry can be None
-    memories = [mem for mem in memories if mem is not None] + [0.0]
-    memory_base = max(memories)


It never really worked robustly. On macOS it almost always errored to open and measure the subprocess

larsoner · 2024-06-29T16:58:19Z

sphinx_gallery/gen_rst.py

+                    f"{gallery_conf['show_memory']=} disabled due to "
+                    f"{gallery_conf['parallel']=}."
+                )
+                gallery_conf["show_memory"] = False


As a dict, gallery_conf` is mutable so the change does not need to be passed back, changing it here changes it everywhere

larsoner · 2024-07-09T19:52:34Z

@lucyleeow okay to merge this now?

lucyleeow · 2024-07-10T01:59:04Z

Yes I'll merge. Sorry I totally missed that you said fixes can be in follow up PR!

Okay I realized that for MNE-Python we treat all warnings in doc builds/examples as errors, so I needed to prevent raising the UserWarning that joblib emits as an error.

Just for info, what UserWarning ?

lucyleeow · 2024-07-10T02:05:34Z

Should we release or should we wait for maybe #1313 ?

jschueller · 2024-07-10T05:13:45Z

Dont wait! (kidding :)
Thanks a lot for pursuing this @larsoner @lucyleeow

larsoner · 2024-07-10T18:46:43Z

It would be good to get #1313 and #1344 then release

larsoner · 2024-07-10T19:53:24Z

Oh and forgot to say:

Just for info, what UserWarning ?

Inside joblib it could emit a warning like "A worker stopped while some jobs were given to the executor", can't remember if it was because some examples executed too quickly or my pre_dispatch was set too high 🤷 But treating that as an error in the doc build (in MNE we treat all uncaught warnings as errors by default) was really problematic because it caused joblib to hang.

jschueller · 2024-07-18T08:27:35Z

still any plans for a new release ? I see PR1313 and PR1344 are done

larsoner · 2024-07-19T14:21:06Z

@lucyleeow you up for making a release soon? If not then I can do it

QuLogic · 2024-07-26T05:36:42Z

sphinx_gallery/gen_rst.py

+        "stale"
+            True if the example was stale.


This doc looks incorrect? On line 1315, the 'stale' key is set to a string of something.

QuLogic · 2024-07-26T07:12:25Z

PS, thanks for getting this done; it's going to speed up our CI a bunch now that we can enable parallel builds.

QuLogic · 2024-07-27T00:51:54Z

Oh and forgot to say:

Just for info, what UserWarning ?

Inside joblib it could emit a warning like "A worker stopped while some jobs were given to the executor", can't remember if it was because some examples executed too quickly or my pre_dispatch was set too high 🤷 But treating that as an error in the doc build (in MNE we treat all uncaught warnings as errors by default) was really problematic because it caused joblib to hang.

It's probably joblib/joblib#883; it restarts the worker if it uses "too much" memory, which is a bit arbitrary. I also had to disable the warning-as-error to get Matplotlib to build; it might be a good idea to document that somewhere.

larsoner · 2024-07-27T14:00:24Z

I also had to disable the warning-as-error to get Matplotlib to build; it might be a good idea to document that somewhere.

I didn't realize other projects also had their own strict filterwarnings("error") in place! In that case yes it would be good to document it somewhere (making clear that it's different from the -W sphinx option).

jschueller mentioned this pull request Oct 28, 2021

Multiprocesor support? #25

Closed

jschueller force-pushed the paral branch 2 times, most recently from 5d2372d to 9baa7d8 Compare October 28, 2021 08:14

jschueller force-pushed the paral branch from 9baa7d8 to 9976c33 Compare October 11, 2022 14:29

Try at parallel gallery generation

d46991f

jschueller force-pushed the paral branch from 9976c33 to d46991f Compare December 2, 2022 14:18

pathos

62dd2e4

jschueller force-pushed the paral branch from bb94ad5 to 62dd2e4 Compare December 4, 2022 16:40

larsoner added 2 commits June 4, 2024 13:49

Merge remote-tracking branch 'upstream/master' into paral

a79163c

FIX: Many fixes

01023a3

larsoner added the enhancement label Jun 4, 2024

larsoner added 4 commits June 4, 2024 17:03

FIX: More

e2a5b8e

FIX: Cleanup

c446f01

FIX: More

4c2247d

FIX: Deps

fc3b8fc

larsoner marked this pull request as ready for review June 5, 2024 00:44

larsoner added 2 commits June 4, 2024 17:49

FIX: Patch

c205cca

FIX: Revert

6916dd7

FIX: Deps

03dd5a7

larsoner changed the title ~~Try at parallel gallery generation~~ ENH: Parallel gallery generation Jun 5, 2024

jschueller commented Jun 5, 2024

View reviewed changes

larsoner added 3 commits June 24, 2024 10:51

DOC: Document considerations

9fee1ce

DOC: Better wording

05caa49

DOC: Better wording

554e567

ENH: Add count

cdc7a93

add ver drop to changes

3cb6893

lucyleeow reviewed Jun 28, 2024

View reviewed changes

larsoner reviewed Jun 29, 2024

View reviewed changes

FIX: Updates

f557d64

lucyleeow merged commit 1b4aecd into sphinx-gallery:master Jul 10, 2024
18 checks passed

jschueller deleted the paral branch July 10, 2024 05:11

larsoner mentioned this pull request Jul 10, 2024

ENH: Parallel example execution in doc build mne-tools/mne-python#12708

Merged

banesullivan mentioned this pull request Jul 17, 2024

Need to speed up the Documentation build in CI pyvista/pyvista#5352

Open

lesteve mentioned this pull request Jul 24, 2024

Consider adding links to Google colab? #411

Open

QuLogic reviewed Jul 26, 2024

View reviewed changes

lesteve mentioned this pull request Jul 26, 2024

Try running examples in parallel during doc build scikit-learn/scikit-learn#29570

Open

lucyleeow mentioned this pull request Aug 1, 2024

DOC add note on filtering joblib warnings #1362

Merged

ENH: Parallel gallery generation #877

ENH: Parallel gallery generation #877

Uh oh!

Conversation

jschueller commented Oct 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

larsoner commented Oct 28, 2021

Uh oh!

larsoner commented Oct 28, 2021

Uh oh!

jschueller commented Jan 11, 2022

Uh oh!

tk0miya commented Jan 11, 2022

Uh oh!

jschueller commented Jan 11, 2022

Uh oh!

larsoner commented Jun 4, 2024

Uh oh!

jschueller commented Jun 5, 2024

Uh oh!

jschueller left a comment

Choose a reason for hiding this comment

Uh oh!

larsoner commented Jun 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jschueller commented Jun 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

larsoner commented Jun 24, 2024

Uh oh!

larsoner commented Jun 24, 2024

MNE-Python

No examples (noplot)

All examples, parallel=1 (serial)

All examples, parallel=4

Uh oh!

lucyleeow commented Jun 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lucyleeow left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

larsoner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

larsoner commented Jul 9, 2024

Uh oh!

lucyleeow commented Jul 10, 2024

jschueller commented Oct 28, 2021 •

edited

Loading

larsoner commented Jun 5, 2024 •

edited

Loading

jschueller commented Jun 5, 2024 •

edited

Loading

lucyleeow commented Jun 27, 2024 •

edited

Loading