feat($core): Improve build performance with Node's worker threads #2189
Conversation
Re-sync master
…om/itsxallwater/vuepress into feat/implement-node-worker-threads
- Make worker thread count a cli option for build - Update docs - Remove verbose mode, use existing silent option - Make worker thread logger respect silent
|
Executing a build of the base VuePress docs w/ 1x worker thread, explicitly requested via CLI.
|
|
Executing a build of the base VuePress docs w/ no worker threads specified via CLI.
|
|
Executing a build of the base VuePress docs w/ 4x worker threads specified via CLI.
|
|
Executing a build of the base VuePress docs w/ 8x worker threads and silent mode specified via CLI.
|
|
As one last example, executing a build of our docs w/ 8x worker threads.
|
|
Officially requesting your thoughts, @kefranabg, since you've been involved in the conversation on #1560 and have your own PR geared towards addressing this. Apologies if I mucked up anything from a contribution guidelines perspective here, or otherwise! Thanks in advance. |
|
@itsxallwater thanks for your PR 👍 Is your project (the one with 1600 pages) public? When I was doing my tests, I was using a project with 600pages and it lasts 1min30s 🤔 To be honest, what makes VuePress build slow is the intensive use of I/O blocking operations. Node.js’s built-in asynchronous I/O operations are more efficient than Workers can be. What I'm trying to say, is that I prefer limiting the use of blocking operation in VuePress and replace them with asynchronous operations (which is the best way to gain performances) instead of using workers |
|
It is, you can check it out at https://github.com/zumasys/docs if you would like. Your explanation makes total sense. FWIW, testing our build with your latest PR updates resulted in a build time of about 25 minutes. |
|
Awesome. I'll use your repos to made some tests. I'll close this PR but I referenced your solution as a workaround in the related issue. |
|
Thanks @kefranabg, I'll be keen to hear how you get along. I'm also happy to take your continued work and test against ours as well. We're building pretty regularly at the moment as we're performing an overall docs migration from another platform, into GitHub + VuePress, so as you might imagine we've got a strong vested interest in tidy build times :) |
|
DataCloneError: Data cannot be cloned, out of memory. My project has about 1500 pages, and I updated it index.js , added worker.js But there was such a error. |
|
@a563905150 did your system truly run out of memory? If so you can limit the memory usage using this command as you build script:
Change 8192 to a value that makes more sense for your system. |
|
@itsxallwater I did, but I still reported an error |
|
What version of node are you running? |
|
@itsxallwater 12.16.1，I found a Worker threads configuration,resourceLimits，https://nodejs.org/docs/latest-v12.x/api/worker_threads.html，but it didn't work |
|
Hmm, I'm rolling 12.14. Do you have a repo I can test against? Have you tried it away from your current environment, perhaps in a CI/CD runner? |
|
@itsxallwater https://github.com/a563905150/openeuler,you can try my repo |
|
@a563905150 It built for me, although two of the docs appear to have some issues that would need to be resolved. Here's the build output: My I tested the build out by adding a new And that Environment wise I ran this in an Ubuntu instance of WSL 2.0. |
|
@itsxallwater Now that it works, the build has been shortened from 37 minutes to 8 minutes, thank you very much! |
|
My pleasure! |
|
Hello guys, somebody working with this cases? I'm trying to improve and fixes my cases with vuepress, my blog has +2K posts and I can´t do build now. On my tests, I tried change some behaviors to try prevent big memory allocation, like set as null some variable missed, split some code to new function to help GC work, but I'm not a performance specialist, then was only random tries. I dont sure but I guess that the renderer can be the issue about the big memory allocation without cleanup. |
Summary
Related to #1560 and is an alternative take on PR 2163. Implement's Node's worker threads to improve build times for larger scale projects. Includes a breaking change in that you must be using a version of Node (10+) that includes the worker threads module.
What kind of change does this PR introduce? (check at least one)
Does this PR introduce a breaking change? (check one)
It is necessary to upgrade your local install of Node to version 10 or greater, when the worker threads module was introduced.
The PR fulfills these requirements:
fix #xxx[,#xxx], where "xxx" is the issue number)You have tested in the following browsers: (Providing a detailed version will be better.)
If adding a new feature, the PR's description includes:
Other information:
The genesis for this PR goes back to this comment within #1560.
Since that comment, I've updated the code to implement the worker thread count as a build option passed to the
buildcommand via-wand I've changed the outputs to conform to the build command's--silentoption, removing the need for an additional "verbose" flag. If the-woption is not invoked, the process will build with 1x worker thread. The cli documentation was updated in this branch to describe these changes.The nature of our project has also changed since that comment. We've cleaned up the documentation down to ~1600 pages but we've increased the number of plugins used, including some that can have a net negative impact to build times like Git Log. For the current state of the project, building with 8x worker threads ran for a total of 4 minutes, whereas running with the base VuePress 1.3 version my last build took ~36 hours. I will post some follow up screenshots illustrating the build testing I performed.