Improvements to webpack tracing, including hot-reload #21652

divmain · 2021-01-29T08:59:35Z

@timneutkens I think this is ready for a review.

I've made some changes to the original design that seem to have paid off. The parenting relationships for traces of normal builds are applied more uniformly, resulting in more intelligible traces:

Hot-reloading is surfaced now, too. I will note, however, that we will want to dig in deeper and find out where the large portion of time at the beginning of hot-reload is spent. Example:

Where did those 180 ms go? At the least, we can now track how long a hot-reload takes, and have a place to start with further investigation.

divmain · 2021-01-29T09:00:46Z

packages/next/build/tracer.ts

@@ -2,6 +2,49 @@ import api, { Span } from '@opentelemetry/api'

 export const tracer = api.trace.getTracer('next', process.env.__NEXT_VERSION)

+const compilerStacks = new WeakMap()
+
+export function stackPush(compiler: any, spanName: string, attrs?: any): any {


Using this approach allowed for easier association between parent and child. The error handling in stackPop down below also helped to identify when the start/stop tracing events were placed incorrectly.

divmain · 2021-01-29T09:01:45Z

packages/next/build/webpack/plugins/profiling-plugin.ts

@@ -14,26 +19,85 @@ function getNormalModuleLoaderHook(compilation: any) {
  return compilation.hooks.normalModuleLoader
 }

+function tracingIsEnabled() {


I found something that works! No need for a top-level span to be present for the plugin to activate.

Should this be moved to tracer.ts so that it can be reused in other places if needed?

ijjk · 2021-01-29T09:10:26Z

Stats from current PR

Default Server Mode (Increase detected ⚠️)

General Overall increase ⚠️

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
buildDuration	10.8s	10.8s	-1ms
nodeModulesSize	75 MB	75 MB	⚠️ +10.9 kB

Page Load Tests Overall increase ✓

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
/ failed reqs	0	0	✓
/ total time (seconds)	2.146	2.045	-0.1
/ avg req/sec	1164.98	1222.49	+57.51
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	1.265	1.288	⚠️ +0.02
/error-in-render avg req/sec	1976.98	1940.43	⚠️ -36.55

Client Bundles (main, webpack, commons)

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
677f882d2ed8..38ed.js gzip	13.1 kB	13.1 kB	✓
framework.HASH.js gzip	39 kB	39 kB	✓
main-HASH.js gzip	6.63 kB	6.63 kB	✓
webpack-HASH.js gzip	751 B	751 B	✓
Overall change	59.4 kB	59.4 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
polyfills-HASH.js gzip	31.3 kB	31.3 kB	✓
Overall change	31.3 kB	31.3 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
_app-2a09aa2..4a98.js gzip	1.28 kB	1.28 kB	✓
_error-8b758..aef6.js gzip	3.46 kB	3.46 kB	✓
hooks-c71ae4..70cd.js gzip	887 B	887 B	✓
index-bbee2f..528b.js gzip	227 B	227 B	✓
link-7faf09b..eba4.js gzip	1.64 kB	1.64 kB	✓
routerDirect..bf84.js gzip	303 B	303 B	✓
withRouter-a..5826.js gzip	302 B	302 B	✓
Overall change	8.09 kB	8.09 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
_buildManifest.js gzip	321 B	321 B	✓
Overall change	321 B	321 B	✓

Rendered Page Sizes

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
index.html gzip	614 B	614 B	✓
link.html gzip	620 B	620 B	✓
withRouter.html gzip	607 B	607 B	✓
Overall change	1.84 kB	1.84 kB	✓

Serverless Mode

General Overall increase ⚠️

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
buildDuration	12.7s	12.8s	⚠️ +14ms
nodeModulesSize	75 MB	75 MB	⚠️ +10.9 kB

Client Bundles (main, webpack, commons)

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
677f882d2ed8..38ed.js gzip	13.1 kB	13.1 kB	✓
framework.HASH.js gzip	39 kB	39 kB	✓
main-HASH.js gzip	6.63 kB	6.63 kB	✓
webpack-HASH.js gzip	751 B	751 B	✓
Overall change	59.4 kB	59.4 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
polyfills-HASH.js gzip	31.3 kB	31.3 kB	✓
Overall change	31.3 kB	31.3 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
_app-2a09aa2..4a98.js gzip	1.28 kB	1.28 kB	✓
_error-8b758..aef6.js gzip	3.46 kB	3.46 kB	✓
hooks-c71ae4..70cd.js gzip	887 B	887 B	✓
index-bbee2f..528b.js gzip	227 B	227 B	✓
link-7faf09b..eba4.js gzip	1.64 kB	1.64 kB	✓
routerDirect..bf84.js gzip	303 B	303 B	✓
withRouter-a..5826.js gzip	302 B	302 B	✓
Overall change	8.09 kB	8.09 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
_buildManifest.js gzip	321 B	321 B	✓
Overall change	321 B	321 B	✓

Serverless bundles

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
_error.js	1 MB	1 MB	✓
404.html	2.67 kB	2.67 kB	✓
hooks.html	1.92 kB	1.92 kB	✓
index.js	1 MB	1 MB	✓
link.js	1.06 MB	1.06 MB	✓
routerDirect.js	1.05 MB	1.05 MB	✓
withRouter.js	1.05 MB	1.05 MB	✓
Overall change	5.19 MB	5.19 MB	✓

Webpack 5 Mode (Increase detected ⚠️)

General Overall increase ⚠️

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
buildDuration	13s	13.1s	⚠️ +99ms
nodeModulesSize	75 MB	75 MB	⚠️ +10.9 kB

Page Load Tests Overall increase ✓

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
/ failed reqs	0	0	✓
/ total time (seconds)	2.095	2.012	-0.08
/ avg req/sec	1193.37	1242.4	+49.03
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	1.351	1.363	⚠️ +0.01
/error-in-render avg req/sec	1851.12	1833.76	⚠️ -17.36

Client Bundles (main, webpack, commons)

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
597-7c719119..27b6.js gzip	13 kB	13 kB	✓
framework.HASH.js gzip	39.3 kB	39.3 kB	✓
main-HASH.js gzip	6.58 kB	6.58 kB	✓
webpack-HASH.js gzip	954 B	954 B	✓
Overall change	59.8 kB	59.8 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
polyfills-HASH.js gzip	31.1 kB	31.1 kB	✓
Overall change	31.1 kB	31.1 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
_app-6e0a903..b885.js gzip	1.26 kB	1.26 kB	✓
_error-7a100..a392.js gzip	3.38 kB	3.38 kB	✓
hooks-f7e199..11b9.js gzip	904 B	904 B	✓
index-3dc22c..ffbb.js gzip	232 B	232 B	✓
link-c010d28..b60a.js gzip	1.63 kB	1.63 kB	✓
routerDirect..323a.js gzip	308 B	308 B	✓
withRouter-a..2ef8.js gzip	304 B	304 B	✓
Overall change	8.02 kB	8.02 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
_buildManifest.js gzip	299 B	299 B	✓
Overall change	299 B	299 B	✓

Rendered Page Sizes

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
index.html gzip	585 B	585 B	✓
link.html gzip	592 B	592 B	✓
withRouter.html gzip	579 B	579 B	✓
Overall change	1.76 kB	1.76 kB	✓

Diffs

Diff for index.html

@@ -43,7 +43,7 @@
         "props": { "pageProps": {} },
         "page": "/",
         "query": {},
-        "buildId": "pTe5shWFjyK2oB9lopKxS",
+        "buildId": "DCNosug1KeykeS-5VlE4b",
         "isFallback": false,
         "gip": true
       }
@@ -77,11 +77,11 @@
       async=""
     ></script>
     <script
-      src="/_next/static/pTe5shWFjyK2oB9lopKxS/_buildManifest.js"
+      src="/_next/static/DCNosug1KeykeS-5VlE4b/_buildManifest.js"
       async=""
     ></script>
     <script
-      src="/_next/static/pTe5shWFjyK2oB9lopKxS/_ssgManifest.js"
+      src="/_next/static/DCNosug1KeykeS-5VlE4b/_ssgManifest.js"
       async=""
     ></script>
   </body>

Diff for link.html

@@ -48,7 +48,7 @@
         "props": { "pageProps": {} },
         "page": "/link",
         "query": {},
-        "buildId": "pTe5shWFjyK2oB9lopKxS",
+        "buildId": "DCNosug1KeykeS-5VlE4b",
         "isFallback": false,
         "gip": true
       }
@@ -82,11 +82,11 @@
       async=""
     ></script>
     <script
-      src="/_next/static/pTe5shWFjyK2oB9lopKxS/_buildManifest.js"
+      src="/_next/static/DCNosug1KeykeS-5VlE4b/_buildManifest.js"
       async=""
     ></script>
     <script
-      src="/_next/static/pTe5shWFjyK2oB9lopKxS/_ssgManifest.js"
+      src="/_next/static/DCNosug1KeykeS-5VlE4b/_ssgManifest.js"
       async=""
     ></script>
   </body>

Diff for withRouter.html

@@ -43,7 +43,7 @@
         "props": { "pageProps": {} },
         "page": "/withRouter",
         "query": {},
-        "buildId": "pTe5shWFjyK2oB9lopKxS",
+        "buildId": "DCNosug1KeykeS-5VlE4b",
         "isFallback": false,
         "gip": true
       }
@@ -77,11 +77,11 @@
       async=""
     ></script>
     <script
-      src="/_next/static/pTe5shWFjyK2oB9lopKxS/_buildManifest.js"
+      src="/_next/static/DCNosug1KeykeS-5VlE4b/_buildManifest.js"
       async=""
     ></script>
     <script
-      src="/_next/static/pTe5shWFjyK2oB9lopKxS/_ssgManifest.js"
+      src="/_next/static/DCNosug1KeykeS-5VlE4b/_ssgManifest.js"
       async=""
     ></script>
   </body>

Commit: 24850bb

timneutkens · 2021-01-29T09:54:20Z

Looks great 🚀

ijjk · 2021-01-29T10:06:27Z

Stats from current PR

Default Server Mode (Increase detected ⚠️)

General Overall increase ⚠️

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
buildDuration	11.8s	11.9s	⚠️ +36ms
nodeModulesSize	75 MB	75 MB	⚠️ +11.2 kB

Page Load Tests Overall increase ✓

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
/ failed reqs	0	0	✓
/ total time (seconds)	2.476	2.484	⚠️ +0.01
/ avg req/sec	1009.69	1006.59	⚠️ -3.1
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	1.704	1.683	-0.02
/error-in-render avg req/sec	1467.2	1485.7	+18.5

Client Bundles (main, webpack, commons)

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
677f882d2ed8..38ed.js gzip	13.1 kB	13.1 kB	✓
framework.HASH.js gzip	39 kB	39 kB	✓
main-HASH.js gzip	6.63 kB	6.63 kB	✓
webpack-HASH.js gzip	751 B	751 B	✓
Overall change	59.4 kB	59.4 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
polyfills-HASH.js gzip	31.3 kB	31.3 kB	✓
Overall change	31.3 kB	31.3 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
_app-2a09aa2..4a98.js gzip	1.28 kB	1.28 kB	✓
_error-8b758..aef6.js gzip	3.46 kB	3.46 kB	✓
hooks-c71ae4..70cd.js gzip	887 B	887 B	✓
index-bbee2f..528b.js gzip	227 B	227 B	✓
link-7faf09b..eba4.js gzip	1.64 kB	1.64 kB	✓
routerDirect..bf84.js gzip	303 B	303 B	✓
withRouter-a..5826.js gzip	302 B	302 B	✓
Overall change	8.09 kB	8.09 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
_buildManifest.js gzip	321 B	321 B	✓
Overall change	321 B	321 B	✓

Rendered Page Sizes

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
index.html gzip	614 B	614 B	✓
link.html gzip	620 B	620 B	✓
withRouter.html gzip	607 B	607 B	✓
Overall change	1.84 kB	1.84 kB	✓

Serverless Mode

General Overall increase ⚠️

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
buildDuration	14.2s	14.3s	⚠️ +42ms
nodeModulesSize	75 MB	75 MB	⚠️ +11.2 kB

Client Bundles (main, webpack, commons)

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
677f882d2ed8..38ed.js gzip	13.1 kB	13.1 kB	✓
framework.HASH.js gzip	39 kB	39 kB	✓
main-HASH.js gzip	6.63 kB	6.63 kB	✓
webpack-HASH.js gzip	751 B	751 B	✓
Overall change	59.4 kB	59.4 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
polyfills-HASH.js gzip	31.3 kB	31.3 kB	✓
Overall change	31.3 kB	31.3 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
_app-2a09aa2..4a98.js gzip	1.28 kB	1.28 kB	✓
_error-8b758..aef6.js gzip	3.46 kB	3.46 kB	✓
hooks-c71ae4..70cd.js gzip	887 B	887 B	✓
index-bbee2f..528b.js gzip	227 B	227 B	✓
link-7faf09b..eba4.js gzip	1.64 kB	1.64 kB	✓
routerDirect..bf84.js gzip	303 B	303 B	✓
withRouter-a..5826.js gzip	302 B	302 B	✓
Overall change	8.09 kB	8.09 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
_buildManifest.js gzip	321 B	321 B	✓
Overall change	321 B	321 B	✓

Serverless bundles

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
_error.js	1 MB	1 MB	✓
404.html	2.67 kB	2.67 kB	✓
hooks.html	1.92 kB	1.92 kB	✓
index.js	1 MB	1 MB	✓
link.js	1.06 MB	1.06 MB	✓
routerDirect.js	1.05 MB	1.05 MB	✓
withRouter.js	1.05 MB	1.05 MB	✓
Overall change	5.19 MB	5.19 MB	✓

Webpack 5 Mode (Increase detected ⚠️)

General Overall increase ⚠️

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
buildDuration	14.6s	14.5s	-93ms
nodeModulesSize	75 MB	75 MB	⚠️ +11.2 kB

Page Load Tests Overall increase ✓

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
/ failed reqs	0	0	✓
/ total time (seconds)	2.426	2.368	-0.06
/ avg req/sec	1030.41	1055.7	+25.29
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	1.693	1.673	-0.02
/error-in-render avg req/sec	1476.44	1494.45	+18.01

Client Bundles (main, webpack, commons)

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
597-7c719119..27b6.js gzip	13 kB	13 kB	✓
framework.HASH.js gzip	39.3 kB	39.3 kB	✓
main-HASH.js gzip	6.58 kB	6.58 kB	✓
webpack-HASH.js gzip	954 B	954 B	✓
Overall change	59.8 kB	59.8 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
polyfills-HASH.js gzip	31.1 kB	31.1 kB	✓
Overall change	31.1 kB	31.1 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
_app-6e0a903..b885.js gzip	1.26 kB	1.26 kB	✓
_error-7a100..a392.js gzip	3.38 kB	3.38 kB	✓
hooks-f7e199..11b9.js gzip	904 B	904 B	✓
index-3dc22c..ffbb.js gzip	232 B	232 B	✓
link-c010d28..b60a.js gzip	1.63 kB	1.63 kB	✓
routerDirect..323a.js gzip	308 B	308 B	✓
withRouter-a..2ef8.js gzip	304 B	304 B	✓
Overall change	8.02 kB	8.02 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
_buildManifest.js gzip	299 B	299 B	✓
Overall change	299 B	299 B	✓

Rendered Page Sizes

	vercel/next.js canary	vercel/next.js webpack-tracing	Change
index.html gzip	585 B	585 B	✓
link.html gzip	592 B	592 B	✓
withRouter.html gzip	579 B	579 B	✓
Overall change	1.76 kB	1.76 kB	✓

Diffs

Diff for index.html

@@ -43,7 +43,7 @@
         "props": { "pageProps": {} },
         "page": "/",
         "query": {},
-        "buildId": "GMfRGO87PZvxCwuByD58D",
+        "buildId": "h2ysPGZ9PVJ6deiBh0C8U",
         "isFallback": false,
         "gip": true
       }
@@ -77,11 +77,11 @@
       async=""
     ></script>
     <script
-      src="/_next/static/GMfRGO87PZvxCwuByD58D/_buildManifest.js"
+      src="/_next/static/h2ysPGZ9PVJ6deiBh0C8U/_buildManifest.js"
       async=""
     ></script>
     <script
-      src="/_next/static/GMfRGO87PZvxCwuByD58D/_ssgManifest.js"
+      src="/_next/static/h2ysPGZ9PVJ6deiBh0C8U/_ssgManifest.js"
       async=""
     ></script>
   </body>

Diff for link.html

@@ -48,7 +48,7 @@
         "props": { "pageProps": {} },
         "page": "/link",
         "query": {},
-        "buildId": "GMfRGO87PZvxCwuByD58D",
+        "buildId": "h2ysPGZ9PVJ6deiBh0C8U",
         "isFallback": false,
         "gip": true
       }
@@ -82,11 +82,11 @@
       async=""
     ></script>
     <script
-      src="/_next/static/GMfRGO87PZvxCwuByD58D/_buildManifest.js"
+      src="/_next/static/h2ysPGZ9PVJ6deiBh0C8U/_buildManifest.js"
       async=""
     ></script>
     <script
-      src="/_next/static/GMfRGO87PZvxCwuByD58D/_ssgManifest.js"
+      src="/_next/static/h2ysPGZ9PVJ6deiBh0C8U/_ssgManifest.js"
       async=""
     ></script>
   </body>

Diff for withRouter.html

@@ -43,7 +43,7 @@
         "props": { "pageProps": {} },
         "page": "/withRouter",
         "query": {},
-        "buildId": "GMfRGO87PZvxCwuByD58D",
+        "buildId": "h2ysPGZ9PVJ6deiBh0C8U",
         "isFallback": false,
         "gip": true
       }
@@ -77,11 +77,11 @@
       async=""
     ></script>
     <script
-      src="/_next/static/GMfRGO87PZvxCwuByD58D/_buildManifest.js"
+      src="/_next/static/h2ysPGZ9PVJ6deiBh0C8U/_buildManifest.js"
       async=""
     ></script>
     <script
-      src="/_next/static/GMfRGO87PZvxCwuByD58D/_ssgManifest.js"
+      src="/_next/static/h2ysPGZ9PVJ6deiBh0C8U/_ssgManifest.js"
       async=""
     ></script>
   </body>

Commit: dc54680

weyert · 2021-01-29T16:05:36Z

Would this allow as a consumer of next.js to get visibility when having a SSR application? I would delighted to be able to get traces to Google Trace via its OpenTracing exporter

Dale Bustad added 12 commits January 27, 2021 15:39

Add granular Webpack tracing

600f161

Add tracing for hot reload.

ae3e542

Create new spans with correct context.

7b0f8c0

Fix typo.

96095b6

Improve intrumentation-detection, move map into class.

e1ff565

Make 'compile' span accessible via spans export.

4064ec7

Fix instrumentation detection.

c664775

Re-parent telemetry events to 'webpack-compile'.

3b38a7d

Use stack-based approach for associating spans.

93becc2

Remove extraneous hot-reloader noise.

31c6e1e

Clean up.

11fc4b7

Use different hook for webpack hot-reload invalidation.

24850bb

divmain requested a review from timneutkens January 29, 2021 08:59

divmain requested review from ijjk, lfades and Timer as code owners January 29, 2021 08:59

ijjk added the type: next label Jan 29, 2021

divmain commented Jan 29, 2021

View reviewed changes

divmain changed the title ~~Webpack tracing~~ Improvements to webpack tracing, including hot-reload Jan 29, 2021

Make sure attributes are generated at the correct time

b57819d

timneutkens approved these changes Jan 29, 2021

View reviewed changes

Merge branch 'canary' into webpack-tracing

dc54680

This comment has been minimized.

Sign in to view

kodiakhq bot merged commit 652a1e8 into canary Jan 29, 2021

kodiakhq bot deleted the webpack-tracing branch January 29, 2021 10:32

snyk-bot mentioned this pull request Dec 16, 2021

[Snyk] Fix for 3 vulnerabilities lethu96/next.js#28

Open

vercel locked as resolved and limited conversation to collaborators Jan 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improvements to webpack tracing, including hot-reload #21652

Improvements to webpack tracing, including hot-reload #21652

divmain commented Jan 29, 2021

divmain Jan 29, 2021 •

edited

Loading

divmain Jan 29, 2021

timneutkens Jan 29, 2021

ijjk commented Jan 29, 2021

Diffs

timneutkens commented Jan 29, 2021

This comment has been minimized.

ijjk commented Jan 29, 2021

Diffs

weyert commented Jan 29, 2021

Improvements to webpack tracing, including hot-reload #21652

Improvements to webpack tracing, including hot-reload #21652

Conversation

divmain commented Jan 29, 2021

divmain Jan 29, 2021 • edited Loading

Choose a reason for hiding this comment

divmain Jan 29, 2021

Choose a reason for hiding this comment

timneutkens Jan 29, 2021

Choose a reason for hiding this comment

ijjk commented Jan 29, 2021

Stats from current PR

Diffs

timneutkens commented Jan 29, 2021

This comment has been minimized.

ijjk commented Jan 29, 2021

Stats from current PR

Diffs

weyert commented Jan 29, 2021

divmain Jan 29, 2021 •

edited

Loading