Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Very high memory usage (~10GB for ~3000 files) #173

@bew

Description

@bew

When the compilation uses A LOT of files, the ManifestPlugin uses A LOT of memory.

As an example, our compilation has ~2800 files (only assets: images, videos, documents), the ManifestPlugin used between 8 & 10GB of RAM to complete. This is way too much!

I've tracked the issue to be at line:
https://github.com/danethurber/webpack-manifest-plugin/blob/bcca8907df46ad6b57199d34d7be876f95170a52/lib/plugin.js#L58
And more specifically, the toJson() call.

This method is used to get easy-to-use object with information about the webpack compilation.
The plugin only uses the assets field of the generated object, here:
https://github.com/danethurber/webpack-manifest-plugin/blob/bcca8907df46ad6b57199d34d7be876f95170a52/lib/plugin.js#L88

The very high memory usage happens while generating the fields related to modules, an easy fix is to disable this part:

-var stats = compilation.getStats().toJson();
+var stats = compilation.getStats().toJson({
+  modules: false,
+});

It'll still generate other data that the plugin doesn't use.
Another fix is to disable everything we don't need:

-var stats = compilation.getStats().toJson();
+var stats = compilation.getStats().toJson({
+  version:     false,
+  hash:        false,
+  timings:     false,
+  builtAt:     false,
+  env:         false,
+  publicPath:  false,
+  outputPath:  false,
+  entrypoints: false,
+  chunks:      false,
+  chunkGroups: false,
+  children:    false,
+  modules:     false, // the memory hungry part is disabled here
+
+  assets: true, // only get assets information
+});

These two diffs resolves the problem, and the compilation uses normal amount of memory.

Let me know what you think about that

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions