Add initial ResponsePayload support (#26938)

Refactors the internals of `next-server` to use `ResponsePayload` instead of `string | null` and manual `sendPayload` calls. This is the first step toward streaming support.

I split `renderToResponseWithComponents` into a separate `renderToResponseWithComponentsInternal` function for ease of review: GitHub's diff rendering was highly misleading, making it seem as though more of the function had changed. The separate function just makes the actual change clearer: we split `renderToHTMLWithComponents` into two promises; one that represents the actual render result, and one that represents all of the work (including background work for e.g. revalidation) that needs to be done as part of generating the result.

These changes make it easier to bubble up a `ResponsePayload`, instead of sometimes calling `sendPayload` out-of-band, centralizing all payload handling in `sendResponse` and eventually a similar function for public APIs that return a string. This centralization will make it much easier to handle a response that needs to be streamed, which is coming soon in another PR.

packages/next/server/dev/next-dev-server.ts

@@ -30,6 +30,7 @@ import Server, {
   WrappedBuildError,
   ServerConstructor,
   FindComponentsResult,
+  ResponsePayload,
 } from '../next-server'
 import { normalizePagePath } from '../normalize-page-path'
 import Router, { Params, route } from '../router'
@@ -636,14 +637,14 @@ export default class DevServer extends Server {
     return await loadDefaultErrorComponents(this.distDir)
   }
 
-  sendHTML(
+  sendResponse(
     req: IncomingMessage,
     res: ServerResponse,
-    html: string
+    response: ResponsePayload
   ): Promise<void> {
     // In dev, we should not cache pages for any reason.
     res.setHeader('Cache-Control', 'no-store, must-revalidate')
-    return super.sendHTML(req, res, html)
+    return super.sendResponse(req, res, response)
   }
 
   protected setImmutableAssetCacheControl(res: ServerResponse): void {