[Memory] Add option to reduce memory usage caused by duplicate strings in webpack-sources #66003

mknichel · 2024-05-20T23:42:51Z

This PR adds a flag to Next.js to enable Webpack options to improve memory usage. See webpack/webpack-sources#155 for a full description of the changes and impact on memory.

This PR adds a patch to webpack-sources temporarily that contains the fixes as the real changes are iterated on to merge upstream in the webpack/webpack-sources repository. After that is done, the patch will be reverted and the latest webpack-sources version will be updated in Next.js.

…es memory usage

…merged

…hel/webpack-sources-memory-reduction

ijjk · 2024-05-22T17:15:47Z

Tests Passed

ijjk · 2024-05-22T17:19:38Z

Stats from current PR

Default Build (Increase detected ⚠️)

General Overall increase ⚠️

	vercel/next.js canary	vercel/next.js mknichel/webpack-sources-memory-reduction	Change
buildDuration	17s	14.5s	N/A
buildDurationCached	8.2s	7.6s	N/A
nodeModulesSize	351 MB	351 MB	⚠️ +9.79 kB
nextStartRea..uration (ms)	409ms	410ms	N/A

Client Bundles (main, webpack)

	vercel/next.js canary	vercel/next.js mknichel/webpack-sources-memory-reduction	Change
3430.HASH.js gzip	169 B	169 B	✓
3601-HASH.js gzip	5.05 kB	5.06 kB	N/A
5450-HASH.js gzip	35.5 kB	35.5 kB	N/A
67a0f7eb-HASH.js gzip	51.4 kB	51.5 kB	N/A
framework-HASH.js gzip	56.5 kB	56.5 kB	N/A
main-app-HASH.js gzip	221 B	215 B	N/A
main-HASH.js gzip	32.3 kB	32.3 kB	N/A
webpack-HASH.js gzip	1.7 kB	1.7 kB	N/A
Overall change	169 B	169 B	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js mknichel/webpack-sources-memory-reduction	Change
polyfills-HASH.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js mknichel/webpack-sources-memory-reduction	Change
_app-HASH.js gzip	193 B	194 B	N/A
_error-HASH.js gzip	192 B	192 B	✓
amp-HASH.js gzip	512 B	511 B	N/A
css-HASH.js gzip	343 B	341 B	N/A
dynamic-HASH.js gzip	2.52 kB	2.53 kB	N/A
edge-ssr-HASH.js gzip	266 B	265 B	N/A
head-HASH.js gzip	363 B	364 B	N/A
hooks-HASH.js gzip	392 B	390 B	N/A
image-HASH.js gzip	4.27 kB	4.27 kB	N/A
index-HASH.js gzip	268 B	268 B	✓
link-HASH.js gzip	2.69 kB	2.69 kB	N/A
routerDirect..HASH.js gzip	327 B	329 B	N/A
script-HASH.js gzip	398 B	398 B	✓
withRouter-HASH.js gzip	323 B	324 B	N/A
1afbb74e6ecf..834.css gzip	106 B	106 B	✓
Overall change	964 B	964 B	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js mknichel/webpack-sources-memory-reduction	Change
_buildManifest.js gzip	482 B	483 B	N/A
Overall change	0 B	0 B	✓

Rendered Page Sizes

	vercel/next.js canary	vercel/next.js mknichel/webpack-sources-memory-reduction	Change
index.html gzip	521 B	519 B	N/A
link.html gzip	535 B	535 B	✓
withRouter.html gzip	517 B	516 B	N/A
Overall change	535 B	535 B	✓

Edge SSR bundle Size

	vercel/next.js canary	vercel/next.js mknichel/webpack-sources-memory-reduction	Change
edge-ssr.js gzip	124 kB	124 kB	N/A
page.js gzip	188 kB	188 kB	N/A
Overall change	0 B	0 B	✓

Middleware size

	vercel/next.js canary	vercel/next.js mknichel/webpack-sources-memory-reduction	Change
middleware-b..fest.js gzip	658 B	656 B	N/A
middleware-r..fest.js gzip	155 B	156 B	N/A
middleware.js gzip	29.6 kB	29.6 kB	N/A
edge-runtime..pack.js gzip	1.02 kB	1.02 kB	✓
Overall change	1.02 kB	1.02 kB	✓

Next Runtimes

	vercel/next.js canary	vercel/next.js mknichel/webpack-sources-memory-reduction	Change
app-page-exp...dev.js gzip	185 kB	185 kB	✓
app-page-exp..prod.js gzip	112 kB	112 kB	✓
app-page-tur..prod.js gzip	123 kB	123 kB	✓
app-page-tur..prod.js gzip	119 kB	119 kB	✓
app-page.run...dev.js gzip	177 kB	177 kB	✓
app-page.run..prod.js gzip	109 kB	109 kB	✓
app-route-ex...dev.js gzip	24.5 kB	24.5 kB	✓
app-route-ex..prod.js gzip	18.3 kB	18.3 kB	✓
app-route-tu..prod.js gzip	18.3 kB	18.3 kB	✓
app-route-tu..prod.js gzip	18.1 kB	18.1 kB	✓
app-route.ru...dev.js gzip	24.2 kB	24.2 kB	✓
app-route.ru..prod.js gzip	18.1 kB	18.1 kB	✓
pages-api-tu..prod.js gzip	9.53 kB	9.53 kB	✓
pages-api.ru...dev.js gzip	9.8 kB	9.8 kB	✓
pages-api.ru..prod.js gzip	9.53 kB	9.53 kB	✓
pages-turbo...prod.js gzip	21.4 kB	21.4 kB	✓
pages.runtim...dev.js gzip	22 kB	22 kB	✓
pages.runtim..prod.js gzip	21.4 kB	21.4 kB	✓
server.runti..prod.js gzip	52.1 kB	52.1 kB	✓
Overall change	1.09 MB	1.09 MB	✓

build cache Overall increase ⚠️

	vercel/next.js canary	vercel/next.js mknichel/webpack-sources-memory-reduction	Change
0.pack gzip	1.67 MB	1.67 MB	⚠️ +1.59 kB
index.pack gzip	129 kB	129 kB	N/A
Overall change	1.67 MB	1.67 MB	⚠️ +1.59 kB

Diff details

Diff for page.js

Diff too large to display

Diff for middleware.js

Diff too large to display

Diff for edge-ssr.js

Diff too large to display

Diff for image-HASH.js

@@ -1,7 +1,7 @@
 (self["webpackChunk_N_E"] = self["webpackChunk_N_E"] || []).push([
   [8358],
   {
-    /***/ 1122: /***/ (
+    /***/ 2058: /***/ (
       __unused_webpack_module,
       __unused_webpack_exports,
       __webpack_require__
@@ -9,7 +9,7 @@
       (window.__NEXT_P = window.__NEXT_P || []).push([
         "/image",
         function () {
-          return __webpack_require__(3294);
+          return __webpack_require__(6302);
         },
       ]);
       if (false) {
@@ -18,7 +18,7 @@
       /***/
     },
 
-    /***/ 2985: /***/ (module, exports, __webpack_require__) => {
+    /***/ 6392: /***/ (module, exports, __webpack_require__) => {
       "use strict";
       /* __next_internal_client_entry_do_not_use__  cjs */
       Object.defineProperty(exports, "__esModule", {
@@ -40,15 +40,15 @@
         __webpack_require__(6024)
       );
       const _head = /*#__PURE__*/ _interop_require_default._(
-        __webpack_require__(7364)
+        __webpack_require__(7174)
       );
-      const _getimgprops = __webpack_require__(3108);
-      const _imageconfig = __webpack_require__(9828);
-      const _imageconfigcontextsharedruntime = __webpack_require__(9053);
-      const _warnonce = __webpack_require__(587);
-      const _routercontextsharedruntime = __webpack_require__(3766);
+      const _getimgprops = __webpack_require__(4804);
+      const _imageconfig = __webpack_require__(2977);
+      const _imageconfigcontextsharedruntime = __webpack_require__(9777);
+      const _warnonce = __webpack_require__(3257);
+      const _routercontextsharedruntime = __webpack_require__(3543);
       const _imageloader = /*#__PURE__*/ _interop_require_default._(
-        __webpack_require__(3189)
+        __webpack_require__(8406)
       );
       // This is replaced by webpack define plugin
       const configEnv = {
@@ -376,7 +376,7 @@
       /***/
     },
 
-    /***/ 3108: /***/ (
+    /***/ 4804: /***/ (
       __unused_webpack_module,
       exports,
       __webpack_require__
@@ -392,9 +392,9 @@
           return getImgProps;
         },
       });
-      const _warnonce = __webpack_require__(587);
-      const _imageblursvg = __webpack_require__(4779);
-      const _imageconfig = __webpack_require__(9828);
+      const _warnonce = __webpack_require__(3257);
+      const _imageblursvg = __webpack_require__(8719);
+      const _imageconfig = __webpack_require__(2977);
       const VALID_LOADING_VALUES =
         /* unused pure expression or super */ null && [
           "lazy",
@@ -769,7 +769,7 @@
       /***/
     },
 
-    /***/ 4779: /***/ (__unused_webpack_module, exports) => {
+    /***/ 8719: /***/ (__unused_webpack_module, exports) => {
       "use strict";
       /**
        * A shared function, used on both client and server, to generate a SVG blur placeholder.
@@ -824,7 +824,7 @@
       /***/
     },
 
-    /***/ 6660: /***/ (
+    /***/ 260: /***/ (
       __unused_webpack_module,
       exports,
       __webpack_require__
@@ -851,10 +851,10 @@
         },
       });
       const _interop_require_default = __webpack_require__(1478);
-      const _getimgprops = __webpack_require__(3108);
-      const _imagecomponent = __webpack_require__(2985);
+      const _getimgprops = __webpack_require__(4804);
+      const _imagecomponent = __webpack_require__(6392);
       const _imageloader = /*#__PURE__*/ _interop_require_default._(
-        __webpack_require__(3189)
+        __webpack_require__(8406)
       );
       function getImageProps(imgProps) {
         const { props } = (0, _getimgprops.getImgProps)(imgProps, {
@@ -886,7 +886,7 @@
       /***/
     },
 
-    /***/ 3189: /***/ (__unused_webpack_module, exports) => {
+    /***/ 8406: /***/ (__unused_webpack_module, exports) => {
       "use strict";
 
       Object.defineProperty(exports, "__esModule", {
@@ -921,7 +921,7 @@
       /***/
     },
 
-    /***/ 3294: /***/ (
+    /***/ 6302: /***/ (
       __unused_webpack_module,
       __webpack_exports__,
       __webpack_require__
@@ -938,8 +938,8 @@
 
       // EXTERNAL MODULE: ./node_modules/.pnpm/react@19.0.0-rc-81c5ff2e04-20240521/node_modules/react/jsx-runtime.js
       var jsx_runtime = __webpack_require__(5293);
-      // EXTERNAL MODULE: ./node_modules/.pnpm/file+..+main-repo+packages+next+next-packed.tgz_react-dom@19.0.0-rc-81c5ff2e04-20240521_react_52riaj2pppk6biljiae3nnywwa/node_modules/next/image.js
-      var next_image = __webpack_require__(6828);
+      // EXTERNAL MODULE: ./node_modules/.pnpm/file+..+diff-repo+packages+next+next-packed.tgz_react-dom@19.0.0-rc-81c5ff2e04-20240521_react_qjsttk5yohmngz5jvpmxy4vvtq/node_modules/next/image.js
+      var next_image = __webpack_require__(3132);
       var image_default = /*#__PURE__*/ __webpack_require__.n(next_image); // CONCATENATED MODULE: ./pages/nextjs.png
       /* harmony default export */ const nextjs = {
         src: "/_next/static/media/nextjs.cae0b805.png",
@@ -969,12 +969,12 @@
       /***/
     },
 
-    /***/ 6828: /***/ (
+    /***/ 3132: /***/ (
       module,
       __unused_webpack_exports,
       __webpack_require__
     ) => {
-      module.exports = __webpack_require__(6660);
+      module.exports = __webpack_require__(260);
 
       /***/
     },
@@ -984,7 +984,7 @@
     /******/ var __webpack_exec__ = (moduleId) =>
       __webpack_require__((__webpack_require__.s = moduleId));
     /******/ __webpack_require__.O(0, [2888, 9774, 179], () =>
-      __webpack_exec__(1122)
+      __webpack_exec__(2058)
     );
     /******/ var __webpack_exports__ = __webpack_require__.O();
     /******/ _N_E = __webpack_exports__;

Diff for main-HASH.js

Diff too large to display

Commit: f6ef8f5

mknichel added 2 commits May 20, 2024 15:46

Add printout for webpack duplicate string memory usage

c06e81d

[Memory] Add experimental option to enable reduction in webpack-sourc…

00cd379

…es memory usage

ijjk added the type: next label May 20, 2024

mknichel changed the title ~~[Memory] Add option to reduce memory usage caused by duplicate strings in webpack-sources"~~ [Memory] Add option to reduce memory usage caused by duplicate strings in webpack-sources May 20, 2024

feedthejim approved these changes May 22, 2024

View reviewed changes

mknichel added 3 commits May 22, 2024 09:59

Add pnpm patched dependency for webpack-sources until upstream PR is …

1868a59

…merged

Add back spurious deletion

f7177f7

Merge branch 'canary' of https://github.com/vercel/next.js into mknic…

38350dc

…hel/webpack-sources-memory-reduction

mknichel added 2 commits May 22, 2024 10:40

Add memory optimizations to webpack build worker

a22eb69

Remove debug memory printout code and revert unnecessary worker changes

60e85b3

mknichel force-pushed the mknichel/webpack-sources-memory-reduction branch from ed9bcf3 to 60e85b3 Compare May 22, 2024 17:52

mknichel marked this pull request as ready for review May 22, 2024 17:52

mknichel requested review from timneutkens, ijjk, shuding, huozhi, ztanner, a team and wyattjoh as code owners May 22, 2024 17:52

Undo formatting for webpack-sources3/index.js

f6ef8f5

ijjk approved these changes May 22, 2024

View reviewed changes

ijjk merged commit 8b360af into canary May 22, 2024
80 of 81 checks passed

ijjk deleted the mknichel/webpack-sources-memory-reduction branch May 22, 2024 20:30

github-actions bot added the locked label Jun 6, 2024

github-actions bot locked as resolved and limited conversation to collaborators Jun 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Memory] Add option to reduce memory usage caused by duplicate strings in webpack-sources #66003

[Memory] Add option to reduce memory usage caused by duplicate strings in webpack-sources #66003

mknichel commented May 20, 2024 •

edited

Loading

ijjk commented May 22, 2024 •

edited

Loading

ijjk commented May 22, 2024 •

edited

Loading

[Memory] Add option to reduce memory usage caused by duplicate strings in webpack-sources #66003

[Memory] Add option to reduce memory usage caused by duplicate strings in webpack-sources #66003

Conversation

mknichel commented May 20, 2024 • edited Loading

ijjk commented May 22, 2024 • edited Loading

Tests Passed

ijjk commented May 22, 2024 • edited Loading

Stats from current PR

mknichel commented May 20, 2024 •

edited

Loading

ijjk commented May 22, 2024 •

edited

Loading

ijjk commented May 22, 2024 •

edited

Loading