{"action":"created","issue":{"url":"https://api.github.com/repos/webpack/webpack/issues/5718","repository_url":"https://api.github.com/repos/webpack/webpack","labels_url":"https://api.github.com/repos/webpack/webpack/issues/5718/labels{/name}","comments_url":"https://api.github.com/repos/webpack/webpack/issues/5718/comments","events_url":"https://api.github.com/repos/webpack/webpack/issues/5718/events","html_url":"https://github.com/webpack/webpack/issues/5718","id":260415063,"node_id":"MDU6SXNzdWUyNjA0MTUwNjM=","number":5718,"title":"Webpack's Performance at Airbnb","user":{"login":"gdborton","id":4172067,"node_id":"MDQ6VXNlcjQxNzIwNjc=","avatar_url":"https://avatars2.githubusercontent.com/u/4172067?v=4","gravatar_id":"","url":"https://api.github.com/users/gdborton","html_url":"https://github.com/gdborton","followers_url":"https://api.github.com/users/gdborton/followers","following_url":"https://api.github.com/users/gdborton/following{/other_user}","gists_url":"https://api.github.com/users/gdborton/gists{/gist_id}","starred_url":"https://api.github.com/users/gdborton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gdborton/subscriptions","organizations_url":"https://api.github.com/users/gdborton/orgs","repos_url":"https://api.github.com/users/gdborton/repos","events_url":"https://api.github.com/users/gdborton/events{/privacy}","received_events_url":"https://api.github.com/users/gdborton/received_events","type":"User","site_admin":false},"labels":[{"id":828350756,"node_id":"MDU6TGFiZWw4MjgzNTA3NTY=","url":"https://api.github.com/repos/webpack/webpack/labels/milestone%20locked%20in","name":"milestone locked in","color":"b60205","default":false},{"id":331503517,"node_id":"MDU6TGFiZWwzMzE1MDM1MTc=","url":"https://api.github.com/repos/webpack/webpack/labels/performance","name":"performance","color":"5319e7","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/webpack/webpack/milestones/15","html_url":"https://github.com/webpack/webpack/milestone/15","labels_url":"https://api.github.com/repos/webpack/webpack/milestones/15/labels","id":2472672,"node_id":"MDk6TWlsZXN0b25lMjQ3MjY3Mg==","number":15,"title":"webpack 4","description":"","creator":{"login":"sokra","id":1365881,"node_id":"MDQ6VXNlcjEzNjU4ODE=","avatar_url":"https://avatars0.githubusercontent.com/u/1365881?v=4","gravatar_id":"","url":"https://api.github.com/users/sokra","html_url":"https://github.com/sokra","followers_url":"https://api.github.com/users/sokra/followers","following_url":"https://api.github.com/users/sokra/following{/other_user}","gists_url":"https://api.github.com/users/sokra/gists{/gist_id}","starred_url":"https://api.github.com/users/sokra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sokra/subscriptions","organizations_url":"https://api.github.com/users/sokra/orgs","repos_url":"https://api.github.com/users/sokra/repos","events_url":"https://api.github.com/users/sokra/events{/privacy}","received_events_url":"https://api.github.com/users/sokra/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":58,"state":"open","created_at":"2017-04-21T16:56:32Z","updated_at":"2018-06-07T06:49:15Z","due_on":"2018-02-24T08:00:00Z","closed_at":null},"comments":78,"created_at":"2017-09-25T21:06:16Z","updated_at":"2018-06-07T06:49:15Z","closed_at":"2018-06-07T06:49:15Z","author_association":"CONTRIBUTOR","body":"**TLDR:** We want to improve webpack's performance in our development environment, and after investigating many performance options we are looking for direction on how to make progress. We want to bring our cached production build and our webpack development server boot times down from 2m58s and 2m28s to ~15 seconds.\r\n\r\n**NOTE:** Please don't close the issue because I deleted the template, if it's important leave a comment and I'll add the template w/ answers to this issue.\r\n\r\n## Some background on our build:\r\n\r\n**We have 4 build configurations**\r\n\r\nName | Description | Chunks | Entrypoints | Modules\r\n--- | --- | --- | --- | ---\r\nClient | Builds our bundles with the browser in mind. | 710 | 587 | 15314\r\nServer | Builds a subset of our client bundles with slightly different ways of handling async loading, externals, etc. | 259 | 259 | 11152\r\nEmbed | Builds a few files meant to be embedded on non airbnb sites (no externals, commons chunks, or code-splitting) | 10 | 10 | 1151\r\nAMP | Builds a few amp specific files to help ensure we're AMP compliant (I'm sure compliant is the wrong term). | 2 | 2 | 1928\r\n\r\n**Current production build timings**\r\n\r\nA full build with no babel/happypack cache and without uglify takes: ~4m31s\r\n\r\nA full build with a cache, but no uglify takes: ~2m58s\r\n\r\n**Timing for restarting the webpack server and loading our homepage**\r\n\r\nThese are measured in our ci system, which is slower than our actual development environment.\r\n\r\n - `Current Master` - full page load, 147673.2 ms (~2m28s)\r\n\r\n## Optimizations that we've investigated\r\n\r\n**Production build optimizations already in place**\r\n - [happypack](https://github.com/amireh/happypack) - This saves us time in our production builds, we parallelize/cache our babel handlebars transforms. We don't use this in our devServer as babel-loader has a built in cache and that is far and away our most expensive loader.\r\n - [webpack-parallel-uglify-plugin](https://github.com/gdborton/webpack-parallel-uglify-plugin) - This parallelizes/caches the uglification process. On a fully cached build this takes ~3 seconds.\r\n\r\n**Development server optimizations already in place**\r\n - We boot our dev server with only a small subset of our build in place as it's unlikely a developer will need to compile all of airbnb.com locally.  We start with any bundles needed for our CommonsChunkPlugin, and add addtional bundles to the compiler as they're needed.  This helps significantly to reduce memory pressure as well as initial boot time, and rebuilds.\r\n - Any server bundles that are added to the server compiler automatically trigger a compile for client side equivalents, it's assumed if you server render something, you'll also want the client code.\r\n \r\n**Build optimizations that help dev and prod**\r\n  - Disabled symlink resolving, this helped both our dev boot times as well as our production build times.\r\n  - babel-loader cache - This is also beneficial for production builds, but is hugely helpful in development as well.\r\n  - `devtool: 'cheap-module-eval-source-map'`\r\n\r\n**Other stuff that we've tried**\r\n\r\nOur primary focus has been on dev server boot up time, as we believe this will have the highest impact on our developers productivity.\r\n\r\n - [hard-source-webpack-plugin](https://github.com/mzgoddard/hard-source-webpack-plugin) - We tried this plugin during its 0.4 releases. It was very promising at the time, shaving 25% off of our dev server boot time. We encountered lots of random caching errors though, and it often resulted in manually deleting the cache and restarting the server to correct.  Also, we recently revisited this plugin and didn't see a performance improvement in our local environment, it's impact may have been dimished by recent webpack performance work.  We also have not revisited this plugin's newer releases.\r\n - [cache-loader](https://github.com/webpack-contrib/cache-loader) - Reduces our dev server boot up time by ~3%. We assume this would have a larger impact without babel-loader's built in cache.\r\n - [thread-loader](https://github.com/webpack-contrib/thread-loader) - Increased our dev server boot up time by ~57%.\r\n - [dll plugin](https://webpack.js.org/plugins/dll-plugin/) - At the time of investigating this was incompatible with the CommonsChunkPlugin, this may have changed. We didn't see much performance improvement here, and didn't think it was worth the risk of development/production build differences.\r\n \r\n**Things we've considered but haven't tried**\r\n\r\n - Disabling CommonsChunkPlugin in development, manually tracking module dependencies for bundles, and caching this information on disk. This would help our boot time as we could cache build results, and verify that they're still correct based on config/module changes.  This comes at a cost though as bundles can't be compiled correctly (matching production) without also compiling with CommonsChunk targets in place.\r\n\r\n**Profiling numbers**\r\n\r\nAgain, these are the times that it would take on average to restart our development server and load up our homepage.\r\n\r\n - `thread-loader` full page load, 232344.2ms (3m52s)\r\n - `cache-loader` full page load, 137157.8ms (2m16s)\r\n - `hard-source-webpack-plugin` full page load, 85980.6ms (1m25s)\r\n \r\n ## Goals\r\n \r\nWhat we REALLY want to see is our webpack development server rebooting and serving requests (with a warmed cache) within 10 seconds. The 10 second number is arbitrary, but I think a good heuristic would be the time it takes to read all the modules from disk + some % of that time.\r\n\r\nLooking over our modules for our client config, it took 1337.9 ms to read in and compute a sha265 hash for each module. If we assume for configuration differences that each of our configs needs to process all of their modules independently (there is major overlap), we can come up with an overall time of 2581ms (1337.9 + 974.3 + 100.6 + 168.4). Let's also assume that we need to read them twice (once for calculating a cacheable hash, and again for reading from cache), and that we're caching source maps so reading from cache will take twice as long. That'll put our build at 7743ms.\r\n\r\nIf we add in some overhead for verifying previous module resolutions, let's say 2 seconds assuming some improvements here based on this [RFC issue](https://github.com/webpack/enhanced-resolve/issues/110). Build would now be at 9743ms.\r\n\r\nOf course we need to account for bundle concatenation, module deduplication, scope hoisting, etc. but I believe that this is not where the bulk of the work is happening.  Let's put another 50% on our build time.\r\n\r\nI believe that **14,614ms** should be a realistic upper limit for a cached build. Probably **20 seconds for a mostly cached build** with a few files changed (this will prevent only optimizing for a full cache hit).\r\n\r\nAfter many profiles, there don't seem to be many items left in terms of low hanging performance improvements.\r\n\r\n**What's the best way for us to contribute to making this a reality?**\r\n\r\n**Additional considerations**\r\n\r\nWe were able to build a tool internally as a proof of concept regarding maximum performance. Without a significant effort investment we were able to configure a simple bundler/server combo that compiled our entire build (incorrectly, so grain of salt) in ~20 seconds.\r\n\r\nWe did this basically by moving module transforming/parsing to worker threads and caching the results of the transforms. The main thread then builds the bundle dependency trees, and sends a sort of manifest to the workers for them to compile the final output.\r\n\r\nI think that this approach would lead to the best performance possible, but it definitely comes with caveats. For one thing, instances cannot be shared amoung workers, so data passed to them must be serializable. "},"comment":{"url":"https://api.github.com/repos/webpack/webpack/issues/comments/395312163","html_url":"https://github.com/webpack/webpack/issues/5718#issuecomment-395312163","issue_url":"https://api.github.com/repos/webpack/webpack/issues/5718","id":395312163,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NTMxMjE2Mw==","user":{"login":"sokra","id":1365881,"node_id":"MDQ6VXNlcjEzNjU4ODE=","avatar_url":"https://avatars0.githubusercontent.com/u/1365881?v=4","gravatar_id":"","url":"https://api.github.com/users/sokra","html_url":"https://github.com/sokra","followers_url":"https://api.github.com/users/sokra/followers","following_url":"https://api.github.com/users/sokra/following{/other_user}","gists_url":"https://api.github.com/users/sokra/gists{/gist_id}","starred_url":"https://api.github.com/users/sokra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sokra/subscriptions","organizations_url":"https://api.github.com/users/sokra/orgs","repos_url":"https://api.github.com/users/sokra/repos","events_url":"https://api.github.com/users/sokra/events{/privacy}","received_events_url":"https://api.github.com/users/sokra/received_events","type":"User","site_admin":false},"created_at":"2018-06-07T06:49:15Z","updated_at":"2018-06-07T06:49:15Z","author_association":"MEMBER","body":"> Seems like this issue has gone quiet since the release of Wepack 4 and that others are reporting performance problems when upgrading from v3 to v4\r\n\r\nSorry we can't watch every discussion in older issues. I better close this one so people will open new issues.\r\n\r\nIf you still have issue with rebuild performance here is a guide how to get it fixed:\r\n\r\n* Rebuilding in production mode is slow. It doesn't use caching. Check if you are using development mode.\r\n* Reduce your configuration to remove all 3rd-party plugins. Is it still slow?\r\n* Don't use a minimizer plugin.\r\n* Try with latest node.js version. And latest webpack version.\r\n* Delete lockfile and node_modules and reinstall.\r\n* Open a new issue.\r\n* If possible create a minimal repro repo which has the same effect.\r\n* If possible capture a cpu profile of the rebuild process and attach it to the issue."}}
{"actor":{"display_login":"sokra"}}