Add handling for repeated slashes #27738

ijjk · 2021-08-03T13:49:59Z

This adds handling for repeated forward/back slashes in Next.js, when these slashes are detected in a request to Next.js we will automatically remove the additional slashes redirecting with a 308 status code which prevents duplicate content when being crawled by search engines.

Fixes: #13011
Fixes: #23772
Closes: #15171
Closes: #25745

…shes

ijjk · 2021-08-03T14:15:29Z

Stats from current PR

Default Build (Increase detected ⚠️)

General Overall increase ⚠️

	vercel/next.js canary	ijjk/next.js add/normalize-slashes	Change
buildDuration	13.6s	13.7s	⚠️ +43ms
buildDurationCached	3.2s	3.1s	-88ms
nodeModulesSize	50.1 MB	50.1 MB	⚠️ +5.12 kB

Page Load Tests Overall decrease ⚠️

	vercel/next.js canary	ijjk/next.js add/normalize-slashes	Change
/ failed reqs	0	0	✓
/ total time (seconds)	2.307	2.328	⚠️ +0.02
/ avg req/sec	1083.75	1073.75	⚠️ -10
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	1.332	1.348	⚠️ +0.02
/error-in-render avg req/sec	1877.45	1854.98	⚠️ -22.47

Client Bundles (main, webpack, commons) Overall increase ⚠️

	vercel/next.js canary	ijjk/next.js add/normalize-slashes	Change
745.HASH.js gzip	179 B	179 B	✓
framework-HASH.js gzip	42.2 kB	42.2 kB	✓
main-HASH.js gzip	22.8 kB	23.1 kB	⚠️ +212 B
webpack-HASH.js gzip	1.5 kB	1.5 kB	✓
Overall change	66.7 kB	67 kB	⚠️ +212 B

Legacy Client Bundles (polyfills)

	vercel/next.js canary	ijjk/next.js add/normalize-slashes	Change
polyfills-HASH.js gzip	31.1 kB	31.1 kB	✓
Overall change	31.1 kB	31.1 kB	✓

Client Pages

	vercel/next.js canary	ijjk/next.js add/normalize-slashes	Change
_app-HASH.js gzip	980 B	980 B	✓
_error-HASH.js gzip	194 B	194 B	✓
amp-HASH.js gzip	312 B	312 B	✓
css-HASH.js gzip	329 B	329 B	✓
dynamic-HASH.js gzip	2.52 kB	2.52 kB	✓
head-HASH.js gzip	350 B	350 B	✓
hooks-HASH.js gzip	904 B	904 B	✓
image-HASH.js gzip	4.13 kB	4.13 kB	✓
index-HASH.js gzip	261 B	261 B	✓
link-HASH.js gzip	1.66 kB	1.66 kB	✓
routerDirect..HASH.js gzip	319 B	319 B	✓
script-HASH.js gzip	387 B	387 B	✓
withRouter-HASH.js gzip	320 B	320 B	✓
bb14e60e810b..30f.css gzip	125 B	125 B	✓
Overall change	12.8 kB	12.8 kB	✓

Client Build Manifests

	vercel/next.js canary	ijjk/next.js add/normalize-slashes	Change
_buildManifest.js gzip	492 B	492 B	✓
Overall change	492 B	492 B	✓

Rendered Page Sizes Overall decrease ✓

	vercel/next.js canary	ijjk/next.js add/normalize-slashes	Change
index.html gzip	531 B	531 B	✓
link.html gzip	543 B	542 B	-1 B
withRouter.html gzip	524 B	523 B	-1 B
Overall change	1.6 kB	1.6 kB	-2 B

Diffs

Diff for main-HASH.js

@@ -1133,12 +1133,7 @@
                 // the asPath unexpectedly e.g. adding basePath when
                 // it wasn't originally present
                 page !== "/404" &&
-                !(
-                  page === "/_error" &&
-                  hydrateProps &&
-                  hydrateProps.pageProps &&
-                  hydrateProps.pageProps.statusCode === 404
-                ) &&
+                page !== "/_error" &&
                 (isFallback ||
                   (data.nextExport &&
                     ((0, _isDynamic).isDynamicRoute(router.pathname) ||
@@ -4423,7 +4418,31 @@
         var urlAsString =
           typeof href === "string"
             ? href
-            : (0, _utils).formatWithValidation(href);
+            : (0, _utils).formatWithValidation(href); // repeated slashes and backslashes in the URL are considered
+        // invalid and will never match a Next.js page/file
+
+        var urlProtoMatch = urlAsString.match(/^[a-zA-Z]{1,}:\/\//);
+        var urlAsStringNoProto = urlProtoMatch
+          ? urlAsString.substr(urlProtoMatch[0].length)
+          : urlAsString;
+        var urlParts = urlAsStringNoProto.split("?");
+
+        if ((urlParts[0] || "").match(/(\/\/|\\)/)) {
+          console.error(
+            "Invalid href passed to next/router: ".concat(
+              urlAsString,
+              ", repeated forward-slashes (//) or backslashes \\ are not valid in the href"
+            )
+          );
+          var normalizedUrl = (0, _utils).normalizeRepeatedSlashes(
+            urlAsStringNoProto
+          );
+          urlAsString = (urlProtoMatch ? urlProtoMatch[0] : "") + normalizedUrl;
+        } // Return because it cannot be routed by the Next.js router
+
+        if (!isLocalURL(urlAsString)) {
+          return resolveAs ? [urlAsString] : urlAsString;
+        }
 
         try {
           base = new URL(
@@ -4433,10 +4452,6 @@
         } catch (_) {
           // fallback to / for invalid asPath values e.g. //
           base = new URL("/", "http://n");
-        } // Return because it cannot be routed by the Next.js router
-
-        if (!isLocalURL(urlAsString)) {
-          return resolveAs ? [urlAsString] : urlAsString;
         }
 
         try {
@@ -6947,6 +6962,7 @@
       exports.getURL = getURL;
       exports.getDisplayName = getDisplayName;
       exports.isResSent = isResSent;
+      exports.normalizeRepeatedSlashes = normalizeRepeatedSlashes;
       exports.loadGetInitialProps = loadGetInitialProps;
       exports.formatWithValidation = formatWithValidation;
       exports.ST = exports.SP = exports.urlObjectKeys = void 0;
@@ -6993,6 +7009,18 @@
         return res.finished || res.headersSent;
       }
 
+      function normalizeRepeatedSlashes(url) {
+        var urlParts = url.split("?");
+        var urlNoQuery = urlParts[0];
+        return (
+          urlNoQuery // first we replace any non-encoded backslashes with forward
+            // then normalize repeated forward slashes
+            .replace(/\\/g, "/")
+            .replace(/\/\/+/g, "/") +
+          (urlParts[1] ? "?".concat(urlParts.slice(1).join("?")) : "")
+        );
+      }
+
       function loadGetInitialProps(_x, _x2) {
         return _loadGetInitialProps.apply(this, arguments);
       }

Diff for index.html

@@ -19,7 +19,7 @@
       defer=""
     ></script>
     <script
-      src="/_next/static/chunks/main-f6943c0106bcf99ed169.js"
+      src="/_next/static/chunks/main-ed99972efba3597ad43c.js"
       defer=""
     ></script>
     <script

Diff for link.html

@@ -19,7 +19,7 @@
       defer=""
     ></script>
     <script
-      src="/_next/static/chunks/main-f6943c0106bcf99ed169.js"
+      src="/_next/static/chunks/main-ed99972efba3597ad43c.js"
       defer=""
     ></script>
     <script

Diff for withRouter.html

@@ -19,7 +19,7 @@
       defer=""
     ></script>
     <script
-      src="/_next/static/chunks/main-f6943c0106bcf99ed169.js"
+      src="/_next/static/chunks/main-ed99972efba3597ad43c.js"
       defer=""
     ></script>
     <script

Webpack 4 Mode (Increase detected ⚠️)

General Overall increase ⚠️

	vercel/next.js canary	ijjk/next.js add/normalize-slashes	Change
buildDuration	10.9s	11s	⚠️ +129ms
buildDurationCached	4.2s	4.2s	-23ms
nodeModulesSize	50.1 MB	50.1 MB	⚠️ +5.12 kB

Page Load Tests Overall increase ✓

	vercel/next.js canary	ijjk/next.js add/normalize-slashes	Change
/ failed reqs	0	0	✓
/ total time (seconds)	2.301	2.316	⚠️ +0.01
/ avg req/sec	1086.33	1079.28	⚠️ -7.05
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	1.337	1.324	-0.01
/error-in-render avg req/sec	1869.56	1888.52	+18.96

Client Bundles (main, webpack, commons) Overall increase ⚠️

	vercel/next.js canary	ijjk/next.js add/normalize-slashes	Change
17.HASH.js gzip	185 B	185 B	✓
677f882d2ed8..HASH.js gzip	13.8 kB	14 kB	⚠️ +196 B
framework.HASH.js gzip	41.9 kB	41.9 kB	✓
main-HASH.js gzip	10.6 kB	10.5 kB	-20 B
webpack-HASH.js gzip	1.19 kB	1.19 kB	✓
Overall change	67.7 kB	67.8 kB	⚠️ +176 B

Legacy Client Bundles (polyfills)

	vercel/next.js canary	ijjk/next.js add/normalize-slashes	Change
polyfills-HASH.js gzip	31.3 kB	31.3 kB	✓
Overall change	31.3 kB	31.3 kB	✓

Client Pages

	vercel/next.js canary	ijjk/next.js add/normalize-slashes	Change
_app-HASH.js gzip	965 B	965 B	✓
_error-HASH.js gzip	3.74 kB	3.74 kB	✓
amp-HASH.js gzip	552 B	552 B	✓
css-HASH.js gzip	333 B	333 B	✓
dynamic-HASH.js gzip	2.71 kB	2.71 kB	✓
head-HASH.js gzip	2.97 kB	2.97 kB	✓
hooks-HASH.js gzip	911 B	911 B	✓
index-HASH.js gzip	231 B	231 B	✓
link-HASH.js gzip	1.64 kB	1.64 kB	✓
routerDirect..HASH.js gzip	298 B	298 B	✓
script-HASH.js gzip	2.94 kB	2.94 kB	✓
withRouter-HASH.js gzip	294 B	294 B	✓
e025d2764813..52f.css gzip	125 B	125 B	✓
Overall change	17.7 kB	17.7 kB	✓

Client Build Manifests

	vercel/next.js canary	ijjk/next.js add/normalize-slashes	Change
_buildManifest.js gzip	499 B	499 B	✓
Overall change	499 B	499 B	✓

Rendered Page Sizes Overall increase ⚠️

	vercel/next.js canary	ijjk/next.js add/normalize-slashes	Change
index.html gzip	577 B	577 B	✓
link.html gzip	589 B	589 B	✓
withRouter.html gzip	569 B	570 B	⚠️ +1 B
Overall change	1.74 kB	1.74 kB	⚠️ +1 B

Diffs

Diff for 677f882d2ed8..c4df.HASH.js

@@ -185,6 +185,7 @@
       exports.getURL = getURL;
       exports.getDisplayName = getDisplayName;
       exports.isResSent = isResSent;
+      exports.normalizeRepeatedSlashes = normalizeRepeatedSlashes;
       exports.loadGetInitialProps = loadGetInitialProps;
       exports.formatWithValidation = formatWithValidation;
       exports.ST = exports.SP = exports.urlObjectKeys = void 0;
@@ -231,6 +232,18 @@
         return res.finished || res.headersSent;
       }
 
+      function normalizeRepeatedSlashes(url) {
+        var urlParts = url.split("?");
+        var urlNoQuery = urlParts[0];
+        return (
+          urlNoQuery // first we replace any non-encoded backslashes with forward
+            // then normalize repeated forward slashes
+            .replace(/\\/g, "/")
+            .replace(/\/\/+/g, "/") +
+          (urlParts[1] ? "?".concat(urlParts.slice(1).join("?")) : "")
+        );
+      }
+
       function loadGetInitialProps(_x, _x2) {
         return _loadGetInitialProps.apply(this, arguments);
       }
@@ -1411,7 +1424,31 @@
         var urlAsString =
           typeof href === "string"
             ? href
-            : (0, _utils).formatWithValidation(href);
+            : (0, _utils).formatWithValidation(href); // repeated slashes and backslashes in the URL are considered
+        // invalid and will never match a Next.js page/file
+
+        var urlProtoMatch = urlAsString.match(/^[a-zA-Z]{1,}:\/\//);
+        var urlAsStringNoProto = urlProtoMatch
+          ? urlAsString.substr(urlProtoMatch[0].length)
+          : urlAsString;
+        var urlParts = urlAsStringNoProto.split("?");
+
+        if ((urlParts[0] || "").match(/(\/\/|\\)/)) {
+          console.error(
+            "Invalid href passed to next/router: ".concat(
+              urlAsString,
+              ", repeated forward-slashes (//) or backslashes \\ are not valid in the href"
+            )
+          );
+          var normalizedUrl = (0, _utils).normalizeRepeatedSlashes(
+            urlAsStringNoProto
+          );
+          urlAsString = (urlProtoMatch ? urlProtoMatch[0] : "") + normalizedUrl;
+        } // Return because it cannot be routed by the Next.js router
+
+        if (!isLocalURL(urlAsString)) {
+          return resolveAs ? [urlAsString] : urlAsString;
+        }
 
         try {
           base = new URL(
@@ -1421,10 +1458,6 @@
         } catch (_) {
           // fallback to / for invalid asPath values e.g. //
           base = new URL("/", "http://n");
-        } // Return because it cannot be routed by the Next.js router
-
-        if (!isLocalURL(urlAsString)) {
-          return resolveAs ? [urlAsString] : urlAsString;
         }
 
         try {

Diff for main-HASH.js

@@ -1680,12 +1680,7 @@ _N_E = (window["webpackJsonp_N_E"] = window["webpackJsonp_N_E"] || []).push([
                 // the asPath unexpectedly e.g. adding basePath when
                 // it wasn't originally present
                 page !== "/404" &&
-                !(
-                  page === "/_error" &&
-                  hydrateProps &&
-                  hydrateProps.pageProps &&
-                  hydrateProps.pageProps.statusCode === 404
-                ) &&
+                page !== "/_error" &&
                 (isFallback ||
                   (data.nextExport &&
                     ((0, _isDynamic).isDynamicRoute(router.pathname) ||

Diff for index.html

@@ -19,11 +19,11 @@
       defer=""
     ></script>
     <script
-      src="/_next/static/chunks/677f882d2ed86fa3467b8979053c1a4c3f8bc4df.a0a7731e0c2053abcc41.js"
+      src="/_next/static/chunks/677f882d2ed86fa3467b8979053c1a4c3f8bc4df.942df9780e1782f25925.js"
       defer=""
     ></script>
     <script
-      src="/_next/static/chunks/main-a62e553cac59bb52acc3.js"
+      src="/_next/static/chunks/main-5dc6f64a20fb6eabbec8.js"
       defer=""
     ></script>
     <script

Diff for link.html

@@ -19,11 +19,11 @@
       defer=""
     ></script>
     <script
-      src="/_next/static/chunks/677f882d2ed86fa3467b8979053c1a4c3f8bc4df.a0a7731e0c2053abcc41.js"
+      src="/_next/static/chunks/677f882d2ed86fa3467b8979053c1a4c3f8bc4df.942df9780e1782f25925.js"
       defer=""
     ></script>
     <script
-      src="/_next/static/chunks/main-a62e553cac59bb52acc3.js"
+      src="/_next/static/chunks/main-5dc6f64a20fb6eabbec8.js"
       defer=""
     ></script>
     <script

Diff for withRouter.html

@@ -19,11 +19,11 @@
       defer=""
     ></script>
     <script
-      src="/_next/static/chunks/677f882d2ed86fa3467b8979053c1a4c3f8bc4df.a0a7731e0c2053abcc41.js"
+      src="/_next/static/chunks/677f882d2ed86fa3467b8979053c1a4c3f8bc4df.942df9780e1782f25925.js"
       defer=""
     ></script>
     <script
-      src="/_next/static/chunks/main-a62e553cac59bb52acc3.js"
+      src="/_next/static/chunks/main-5dc6f64a20fb6eabbec8.js"
       defer=""
     ></script>
     <script

Commit: 965d14b

packages/next/server/next-server.ts

packages/next/shared/lib/router/router.ts

This adds handling for repeated forward/back slashes in Next.js, when these slashes are detected in a request to Next.js we will automatically remove the additional slashes redirecting with a 308 status code which prevents duplicate content when being crawled by search engines. Fixes: vercel#13011 Fixes: vercel#23772 Closes: vercel#15171 Closes: vercel#25745

Add handling for repeated slashes

56a08be

ijjk added created-by: Next.js team PRs by the Next.js team. type: next labels Aug 3, 2021

ijjk added 2 commits August 3, 2021 08:57

fix-lint

3607162

Merge remote-tracking branch 'upstream/canary' into add/normalize-sla…

89f2761

…shes

This comment has been minimized.

Sign in to view

re-use util

965d14b

ijjk marked this pull request as ready for review August 3, 2021 14:44

ijjk requested review from huozhi, padmaia, shuding, styfle and timneutkens as code owners August 3, 2021 14:44

styfle reviewed Aug 3, 2021

View reviewed changes

packages/next/server/next-server.ts Show resolved Hide resolved

styfle reviewed Aug 3, 2021

View reviewed changes

packages/next/server/next-server.ts Show resolved Hide resolved

styfle reviewed Aug 3, 2021

View reviewed changes

packages/next/shared/lib/router/router.ts Show resolved Hide resolved

ijjk requested a review from styfle August 3, 2021 15:04

styfle approved these changes Aug 3, 2021

View reviewed changes

kodiakhq bot merged commit 4ab41ef into vercel:canary Aug 3, 2021

ijjk deleted the add/normalize-slashes branch August 3, 2021 15:06

azu mentioned this pull request Aug 16, 2021

2021-08-17のJS: Next.js 11.1、Rails 7で予定されているimport mapsベースのJSライブラリの利用 jser/jser.github.io#903

Merged

vercel locked as resolved and limited conversation to collaborators Jan 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add handling for repeated slashes #27738

Add handling for repeated slashes #27738

ijjk commented Aug 3, 2021 •

edited

Loading

This comment has been minimized.

ijjk commented Aug 3, 2021

Diffs

Diffs

Add handling for repeated slashes #27738

Add handling for repeated slashes #27738

Conversation

ijjk commented Aug 3, 2021 • edited Loading

This comment has been minimized.

ijjk commented Aug 3, 2021

Stats from current PR

Diffs

Diffs

ijjk commented Aug 3, 2021 •

edited

Loading