Webpack 4 chunkhash/contenthash can vary between builds #7179

mikeyoon · 2018-05-02T21:37:39Z

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
I'm finding that the chunkhash varies between builds in v4. Using the diff utility between the the different files shows no differences. Attempting to narrow the codebase down reduces the frequency of different hashes.

If the current behavior is a bug, please provide the steps to reproduce.
I've been unsuccessful in creating a small test case as it's just too inconsistent and would take such an enormous amount of time for me to figure out by slicing and dicing our codebase.

I'm happy to help as much as I can to find a repro, but I'd need some information about diagnostics I can turn on to figure out what's happening.

What is the expected behavior?
The chunkhash should be consistent, as does in version 3.

If this is a feature request, what is motivation or use case for changing the behavior?

Please mention other relevant information such as the browser version, Node.js version, webpack version, and Operating System.

Webpack 4.6.0, Node 8.9.4, OS X 10.13.4

Here's our webpack.config.js

const path = require('path');
const webpack = require('webpack');
const ProgressBarPlugin = require('progress-bar-webpack-plugin');
const childProcess = require('child_process');
const UglifyJsPlugin = require('uglifyjs-webpack-plugin')
const AssetsPlugin = require('assets-webpack-plugin')
const CleanWebpackPlugin = require('clean-webpack-plugin')
const WebpackBuildNotifierPlugin = require('webpack-build-notifier');

const root = path.resolve(__dirname, './src');
const webappPath = path.resolve(__dirname, '..');
const deployPath = path.resolve(webappPath, './public/assets');

const dev = !process.env.NODE_ENV;
const ci = process.env.NODE_ENV === 'ci';
const prod = process.env.NODE_ENV === 'production';

if (prod) {
  console.log('PRODUCTION MODE ENABLED');
}

// don't hash the filename unless we're building for prod or staging. Hashed filenames interfere with karma tests.
const outputFilename = dev || ci ? '[name].bundle.js' : '[name].[chunkhash].bundle.js';
const excludedMapSources = dev ? [/vendor.*bundle.js/, /manifest.*bundle.js/] : [/manifest.*bundle.js/];

function getPlugins() {
  let plugins = [
    new CleanWebpackPlugin(['public/assets/*'], {
      allowExternal: true,
      root: webappPath
    }),

    new webpack.SourceMapDevToolPlugin({
      filename: "[file].map",
      exclude: excludedMapSources
    }),

    new AssetsPlugin({
      includeManifest: true,
      path: deployPath,
      prettyPrint: true
    })
  ];

  plugins.push(new webpack.NamedModulesPlugin());
  plugins.push(new webpack.optimize.ModuleConcatenationPlugin());

  return plugins;
}

module.exports = {
  context: root,

  mode: prod ? 'production' : 'development',
  devtool: false, // rely on SourceMapDevToolPlugin
  cache: dev,

  stats: {
    chunks: false,
    chunkModules: false,
    entrypoints: false,
    modules: false,
    version: false,
    hash: false,
    timings: false,
  },

  entry: {
    app: './core-app/bootstrap.ts',
    embed: './core-embed/bootstrap.ts'
  },

  watchOptions: {
    aggregateTimeout: 300,
    ignored: /node_modules/
  },

  resolve: {
    extensions: ['.ts', '.tsx', '.js', '.jsx'],
    alias: {
      handsontable: 'handsontable/dist/handsontable.full.js'
    }
  },

  output: {
    filename: outputFilename,
    path: path.resolve(__dirname, '../public/assets'),
  },

  optimization: {
    runtimeChunk: {
      name: "manifest",
    },
    splitChunks: {
      cacheGroups: {
        // default: false,
        commons: {
          test: /[\\/]node_modules[\\/]/,
          name: "vendor",
          chunks: "initial",
          enforce: true
        }
      },
    },
  },

  externals: {
    nvd3: 'window.nv',
    highcharts: "window.Highcharts",
    angular: 'window.angular',
    jquery: 'window.jQuery',
    lodash: 'window._',
    moment: 'window.moment',
    ace: 'window.ace',
    d3: 'window.d3',
    chiliPiper: 'window.ChiliPiper',
  },

  plugins: getPlugins(),

  module: {
    rules: [
      {
        test: /\.ts(x?)$/,
        exclude: [/\.(spec|e2e)\.ts$/],
        use: [{
          loader: 'ts-loader',
          options: {
            experimentalWatchApi: true // seems to work ok atm, but performance improvement varies
          }
        }]
      }, {
        test: /\.(html|css)$/,
        loader: 'raw-loader',
        exclude: /\.async\.(html|css)$/
      }, {
        test: /\.svg$/,
        loader: 'url-loader?' + JSON.stringify({
          name: '[name]_[hash]'
        })
      }
    ]
  }
};

michael-ciniawsky · 2018-05-02T21:50:29Z

- const outputFilename = dev || ci ? '[name].bundle.js' : '[name].[chunkhash].bundle.js';
+ const outputFilename = dev || ci ? '[name].bundle.js' : '[name].[contenthash].bundle.js';

TLDR: Use [contenthash] instead of [chunkhash] to ensure that every asset (output file) has a stable hash based on it's contents (only).

[chunkhash] isn't very 'stable' as it can change when e.g a module of a chunk is moved for some reason etc etc (It's always hard to tell why and will always be based on how [chunkhash] works (in terms of browser caching)). For predictable long term (browser) caching the contents of the asset are the only reliable hashing source since browsers cache files based on Map { url => content } relations without any knowledge of webpack chunks or the like...

michael-ciniawsky · 2018-05-02T21:56:14Z

Also don't forget to set the chunkFilename for Code Splitting (Async Chunks)

const output = {
  filename: dev || ci ? '[name].bundle.js' : '[name].[contenthash].bundle.js',
  chunkFilename: dev || ci ? '[name].chunk.js' : '[name].[contenthash].chunk.js'
}

mikeyoon · 2018-05-02T22:46:40Z

I changed it to contenthash, but I still get different hashes and they have the same content. I even took out the sourcemap output to make sure they are exactly the same.

viktorlarsson · 2018-05-18T13:41:47Z

Experiencing the same issue, did you resolve it?

EDIT: This plugin fixed my issue:
https://www.npmjs.com/package/webpack-plugin-hash-output

alexander-akait · 2018-05-18T13:50:26Z

@mikeyoon can you create minimum reproducible test repo?

mikeyoon · 2018-05-18T17:45:11Z

@evilebottnawi I tried, but wasn't able to narrow it down because I don't know what causes the issue or how the logic works.

alexander-akait · 2018-05-18T17:53:11Z

@mikeyoon can we close issue? if problem again appear we can reopen

mikeyoon · 2018-05-18T17:56:35Z

The issue has already been closed earlier, although it definitely still happens to me and I have chosen not to upgrade to Webpack 4 because of it. I don't really care too much at this point as v3 still works for us.

MQuy · 2018-05-18T21:31:21Z

it happens for me too and I do use contenthash but they are changed per my build 😢 (it seems that chunk's content is different per build)

jacobwindsor · 2018-06-14T17:36:19Z

I also have this issue with chunkhash but I am seeing a pattern:

Only the chunkhash of one "vendor" splitChunk is affected
Only the hash is changed and not the file
- I am using sourcemaps so the file does change but only on the line referencing the sourcemap
The hash remains consistent through builds started in the same bash
- I.e. if I run the build several times from git bash it's all good. Only if I run the build from VS integrated terminal and then from git bash is there a difference.

ArtemZag · 2018-08-06T10:33:22Z

Probably the hash is different because it's based on module IDs that are generated during the build.

tkh44 · 2018-08-20T19:22:00Z

This section of the docs might be relevant.

... we can see that all three have. This is because each module.id is incremented based on resolving order by
default. Meaning when the order of resolving is changed, the IDs will be changed as well. So, to recap:

The main bundle changed because of its new content.
The vendor bundle changed because its module.id was changed.
And, the manifest bundle changed because it now contains a reference to a new module.
The first and last are expected -- it's the vendor hash we want to fix. Luckily, there are two plugins we can use
to resolve this issue. The first is the NamedModulesPlugin, which will use the path to the module rather than a
numerical identifier. While this plugin is useful during development for more readable output, it does take a bit
longer to run. The second option is the HashedModuleIdsPlugin, which is recommended for production builds:

Now, despite any new local dependencies, our vendor hash should stay consistent between builds:

https://webpack.js.org/guides/caching/#module-identifiers

mikeyoon · 2018-08-21T19:54:59Z

I have HashedModuleIdsPlugin enabled, so there is probably some other factor.

ArtemZag · 2018-08-22T09:04:26Z

@mikeyoon , try also to enable NamedChunksPlugin (otherwise your chunks will get incremental ids as well as the modules)

mikeyoon · 2018-08-24T18:56:44Z

Didn't work with [contenthash] or [chunkhash] unfortunately. There's definitely something with our codebase that's triggering the issue. Not all of our chunks have inconsistent hashes.

jakub-g · 2018-09-28T15:33:52Z

FYI
We had the same issue in our builds (though with webpack 3) when using chunkhash and I tracked it down to script-loader.
webpack-contrib/script-loader#56

I think plugins and loaders (raw-loader) are most likely to contribute to the issue.
Right now I'm in the process of migration to webpack 4 and contenthash, will report my findings in the coming weeks.

jakub-g · 2018-12-22T11:30:41Z

Update: from what I've noticed in the recent weeks, the issue actually happens also for some new chunks in our project that are not loaded via script-loader. The issue is not script-loader then (probably).

On my machine, for a given git revision, contenthashes are deterministic, but when running a different git revision, some output chunks are often identical, but contenthash is different than a few days before.

It seems that the source of this is some internal webpack global build state, and that contenthash is not a reliable tool for the job, as of webpack4. I will try webpack5 in the coming weeks to see if it improves anything. If not, an alternative would be some hashing tool independent of webpack (after-build), as contenthash is also not taking into account the minification process (webpack-contrib/terser-webpack-plugin#18 (comment)), and I suspect many other webpack plugins/loaders may have the same issue.

onlybye · 2019-03-28T07:26:56Z

Didn't work with [contenthash] or [chunkhash] unfortunately. There's definitely something with our codebase that's triggering the issue. Not all of our chunks have inconsistent hashes.

@mikeyoon have you solved this problem ?

alexander-akait · 2019-03-28T09:38:25Z

Please create minimum reproducible test repo

mikeyoon · 2019-03-30T03:29:26Z

I have not looked at the issue since I posted about it last time. I also haven't had a chance to try out the latest major release either. We're still just using v3.x for now.

advl · 2019-05-13T23:58:48Z

Having the same issue on webpack 4, both with contenthash and chunkhash. If I find some time I will make a minimum repo. Shall the ticket be reopened ?

As mentioned in the below link, [chunkhash] is unstable. webpack/webpack#7179 (comment) To ensure that every bundled files have consistent hash names based on it's contents, use [contenthash] instead of [chunkhash].

halfnibble · 2019-06-27T20:44:31Z

We are seeing this problem too. Unfortunately, it's inconsistent and the codebase is private. Could it have something to do with running 10+ webpack commands simultaneously in different threads? (We are building a mono repo managed by rush.

My assigned task is to determine why this is inconsistent, and to fix it, so any help you can provide would be much appreciated. Thanks.

vcraescu · 2019-10-29T07:48:36Z

Same issue here. We do async imports and because of this we can't run the app on multiple machines because the code is built separately on each machine so generated files names are not the same on all the machines.

halfnibble · 2019-10-29T18:07:27Z

I should have come back and posted an update.

Short answer, Webpack's hashing algorithm is deterministic. But the content being sent to it may not be. I discovered a whole slew of issues:

Some npm packages create absolute paths in postInstall scripts.
Some packages' error messages get sent to the hashing algorithm containing an absolute path.
And we had one webpack loader generating content containing absolute paths.

I found the easiest way to debug this issue was to override the webpack hashing algorithm and add a little script to detect for my local absolute path. Here is approximately what I used:
https://gist.github.com/halfnibble/c9f0c3b8d61e7002c799b5d24ae05643

And then in webpack config:

{
  ...
  output: {
    hashFunction: HashFunction,
    ...
  }
}

alexander-akait · 2019-11-05T13:13:13Z

@halfnibble can you create minimum reproducible test repo, maybe we can fix some cases on webpack side

halfnibble · 2019-11-05T16:33:37Z

@evilebottnawi As far as I can tell, there is absolutely nothing wrong with webpack's hashing algorithm. The problem lies in certain npm packages and webpack loaders. None of the problems I encountered were in code from the webpack organization.

I don't know if it would be worthwhile to create a content hash debug feature like the one we developed to pin point the packages and loaders causing problems.

alexander-akait · 2019-11-05T16:59:15Z

@halfnibble thanks for feedback, yep, some loaders/plugins emit assets with absolute path, better open issues in repos with loaders/plugins

jakub-g · 2019-11-06T17:18:44Z

@halfnibble re: #7179 (comment)

fantastic comment, thanks to it I revisited the topic and found that it was indeed script-loader that was breaking hashing in our project (due to the usage of path.join and require.resolve which both output absolute paths). I migrated away from it to raw-loader instead, and submitted the PR to deprecate script-loader.

The info about not using absolute paths is documented in "writing a loader" doc:

https://webpack.js.org/contribute/writing-a-loader/#absolute-paths

As mentioned in the below link, [chunkhash] is unstable. webpack/webpack#7179 (comment) To ensure that every bundled files have consistent hash names based on it's contents, use [contenthash] instead of [chunkhash].

michael-ciniawsky closed this as completed May 2, 2018

mikeyoon changed the title ~~Webpack 4 chunkhash can vary between builds~~ Webpack 4 chunkhash/contenthash can vary between builds May 3, 2018

alexander-akait added the X2: answer required label May 18, 2018

This was referenced Dec 18, 2018

Trying [contentHash] instead of [chunkHash] facebook/create-react-app#6063

Closed

Trying [contentHash] instead of [chunkHash] joshnabbott/create-react-app#1

Merged

jakub-g mentioned this issue Dec 22, 2018

Using script-loader seems to lead to a different [chunkhash] being generated each time webpack-contrib/script-loader#56

Closed

thallada mentioned this issue Dec 31, 2018

Include absoluteRuntime in babel preset docs facebook/create-react-app#5847

Merged

rokoroku mentioned this issue Jan 25, 2019

Fix for filename/chunkFilename error rokoroku/react-redux-typescript-boilerplate#63

Merged

tatsuyafw mentioned this issue May 16, 2019

Use [contenhash] instead of [chunkhash] for JavaScript rails/webpacker#2094

Merged

gabiudrescu mentioned this issue Oct 29, 2019

Add Webpack Encore Sylius/Sylius#10803

Merged

michal-bednarz mentioned this issue Nov 12, 2019

404 during loading chunks when using contenthash #9961

Closed

StarpTech mentioned this issue Nov 20, 2019

Contenthash of commons chunks differs on different machines vercel/next.js#9469

Closed

rtercias mentioned this issue Jul 22, 2020

Added support for non-square icon sizes arthurbergmz/webpack-pwa-manifest#130

Open

billfienberg mentioned this issue Sep 28, 2020

add medallia styling as separate output department-of-veterans-affairs/vets-website#14312

Merged

2 tasks

alexandernanberg mentioned this issue Nov 2, 2020

Absolute path in loader.js breaks Webpack contenthash jantimon/html-webpack-plugin#1548

Closed

jscheid mentioned this issue Jan 4, 2021

Revisit WSI<->HWP integration jantimon/html-webpack-plugin#1554

Closed

niklasnatter mentioned this issue Mar 3, 2021

Error when building admin with custom JS: TypeError: (void 0) is not a function sulu/sulu#5853

Closed

nytai mentioned this issue Jun 2, 2021

build(webpack): use [contenthash] instead of [chunkhash] apache/superset#14942

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Webpack 4 chunkhash/contenthash can vary between builds #7179

Webpack 4 chunkhash/contenthash can vary between builds #7179

mikeyoon commented May 2, 2018

michael-ciniawsky commented May 2, 2018 •

edited

Loading

michael-ciniawsky commented May 2, 2018 •

edited

Loading

mikeyoon commented May 2, 2018

viktorlarsson commented May 18, 2018 •

edited

Loading

alexander-akait commented May 18, 2018

mikeyoon commented May 18, 2018

alexander-akait commented May 18, 2018 •

edited

Loading

mikeyoon commented May 18, 2018

MQuy commented May 18, 2018 •

edited

Loading

jacobwindsor commented Jun 14, 2018

ArtemZag commented Aug 6, 2018 •

edited

Loading

tkh44 commented Aug 20, 2018

mikeyoon commented Aug 21, 2018

ArtemZag commented Aug 22, 2018 •

edited

Loading

mikeyoon commented Aug 24, 2018

jakub-g commented Sep 28, 2018

jakub-g commented Dec 22, 2018

onlybye commented Mar 28, 2019

alexander-akait commented Mar 28, 2019

mikeyoon commented Mar 30, 2019

advl commented May 13, 2019

halfnibble commented Jun 27, 2019

vcraescu commented Oct 29, 2019

halfnibble commented Oct 29, 2019

alexander-akait commented Nov 5, 2019

halfnibble commented Nov 5, 2019

alexander-akait commented Nov 5, 2019

jakub-g commented Nov 6, 2019

Webpack 4 chunkhash/contenthash can vary between builds #7179

Webpack 4 chunkhash/contenthash can vary between builds #7179

Comments

mikeyoon commented May 2, 2018

michael-ciniawsky commented May 2, 2018 • edited Loading

michael-ciniawsky commented May 2, 2018 • edited Loading

mikeyoon commented May 2, 2018

viktorlarsson commented May 18, 2018 • edited Loading

alexander-akait commented May 18, 2018

mikeyoon commented May 18, 2018

alexander-akait commented May 18, 2018 • edited Loading

mikeyoon commented May 18, 2018

MQuy commented May 18, 2018 • edited Loading

jacobwindsor commented Jun 14, 2018

ArtemZag commented Aug 6, 2018 • edited Loading

tkh44 commented Aug 20, 2018

mikeyoon commented Aug 21, 2018

ArtemZag commented Aug 22, 2018 • edited Loading

mikeyoon commented Aug 24, 2018

jakub-g commented Sep 28, 2018

jakub-g commented Dec 22, 2018

onlybye commented Mar 28, 2019

alexander-akait commented Mar 28, 2019

mikeyoon commented Mar 30, 2019

advl commented May 13, 2019

halfnibble commented Jun 27, 2019

vcraescu commented Oct 29, 2019

halfnibble commented Oct 29, 2019

alexander-akait commented Nov 5, 2019

halfnibble commented Nov 5, 2019

alexander-akait commented Nov 5, 2019

jakub-g commented Nov 6, 2019

michael-ciniawsky commented May 2, 2018 •

edited

Loading

michael-ciniawsky commented May 2, 2018 •

edited

Loading

viktorlarsson commented May 18, 2018 •

edited

Loading

alexander-akait commented May 18, 2018 •

edited

Loading

MQuy commented May 18, 2018 •

edited

Loading

ArtemZag commented Aug 6, 2018 •

edited

Loading

ArtemZag commented Aug 22, 2018 •

edited

Loading