DOC: Enable parallel builds #28617
DOC: Enable parallel builds #28617
Conversation
|
It looks like we are hitting joblib/joblib#883 (comment) which we could patch as in the link, but the deadlock is actually because we set warnings to be errors, so I've set |
The latest sphinx-gallery 0.17.0 adds support for parallel building of examples, and the issue in pydata-sphinx-theme was fixed in 0.15.4, so we can try enabling it again.
|
It also looks like we didn't really use any more RAM (and the peak is before we even start building docs), so I'm going to try bumping up to 4 processes. |
|
Even at 4 jobs, we didn't really use much more RAM (max 53%), and didn't really get better times (15m16s for 2 vs 14m17s for 4). I think also I should check what happens in release mode just to be sure that that won't OOM. I'll set to draft just so that this doesn't get merged while that is in progress. |
|
Results:
(Note that the maximums are from the Resources tab after the first 2 minutes; I believe the spike at 2min is from earlier job steps, not the build step.) So for non-release (PR) builds, 2x is ~7min quicker or ~68% of the previous time, 4x is ~8min quicker or ~64% of the previous time. For release (main) builds, 4x is ~25min quicker or ~45% of the previous time. I didn't bother checking release builds with 2x, since we didn't get anywhere close to OOMing. I'm going to remove the release-job commit, and mark as ready again. |
#28617 introduced parallel gallery builds. This needs joblib as a dependency for sphinx-gallery. `requirements.txt` solves this via `sphinx-gallery[parallel]>=0.12.0`. Unfortunately, conda does not support variants, therefore we have to add the dependency explicitly to `environment.yml`.
#28617 introduced parallel gallery builds. This needs joblib as a dependency for sphinx-gallery. `requirements.txt` solves this via `sphinx-gallery[parallel]>=0.12.0`. Unfortunately, conda does not support variants, therefore we have to add the dependency explicitly to `environment.yml`.
PR summary
The latest sphinx-gallery 0.17.0 adds support for parallel building of examples, and the issue in pydata-sphinx-theme was fixed in 0.15.4, so we can try enabling it again.
We're on a large docker instance, with 4 cores and 8GB RAM; the RAM tracking shows we use max 43% / average 37%, so I've set parallelism to 2, which hopefully won't OOM.
cf #28244 for the last attempt that was stalled by the pydata-sphinx-theme issue.
PR checklist