Are chunk hashes not deterministic based on BUILD_ID?

[CaptainN]

Summary

I have a site that we have been running with 2 horizontally scaled pods, which uses generateBuildId to return the git hash of the current commit for reliable scaling. I thought that meant the chunk hashes would be deterministically generated - that is, they'd be the same between pods (which are built at startup in the runtime environment, since static generation occurs at build time, and the necessary services are not available in the build environment).

This worked for a long time, but we just had a site meltdown because the chunk hash names didn't match, and when balancing occured, it was causing errors in loading. The solution was to scale back down to one instance for the time being. I did confirm by inspecting the build artifacts that .next/BUILD_ID matched between the pods, but the chunk names which were erroring did not.

Is BUILD_ID not intended to be used this way?

Additional information

I'm currently on next@14.1.4

Operating System:
  Platform: darwin (dev machine - we run node:20-alpine in production)
  Arch: arm64
  Version: Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000
Binaries:
  Node: 20.11.1
  npm: 10.2.4 (this project uses npm)
  Yarn: 1.22.22
  pnpm: 8.15.4
Relevant Packages:
  next: 14.1.4
  eslint-config-next: 14.1.3
  react: 18.2.0
  react-dom: 18.2.0
  typescript: 5.4.2
Next.js Config:
  output: N/A

Example

No response

[CaptainN]

I just did a quick test, and can confirm, the chunk names are not the same between builds. All I did is build once locally, delete the .next folder, then build again. The results are different chunk names:

# First Build:
❯ cat BUILD_ID
3cdb0cdf731951b8f03f4d8c16c600c5ead0d6eb

# in:
/.next/static/chunks/app/about/methodology/[slug]

❯ ls
page-fc743ef5cdb60c1d.js

# Second Build:
❯ cat BUILD_ID
3cdb0cdf731951b8f03f4d8c16c600c5ead0d6eb

# in:
/.next/static/chunks/app/about/methodology/[slug]

❯ ls
page-8acd4de2256ef601.js

Many of the build artifacts do have the same names - it seems to be these app chunks that don't get the same (deterministic) hashes.

[CaptainN]

Is this expected behavior, or is there a bug?

[CaptainN]

(Clearly a bug.)

[CaptainN]

bump

[Djay77]

Hey,

We have exactly the same problem here !
Multiples pods who build at startup and generate different chunks => some JS are in 404, then the page fall in error

What is the real utility of the Next.js config generateBuildId ... ?

[Djay77]

We have to override the webpack config in next.config file

webpack: (
        config,
        { buildId }
    ) => {
        config.output.filename = config.output.filename.replace('[chunkhash]', buildId);

        return config
    },

We have now the same name generated for our files BUT the content change

[CaptainN]

I tried to create an issue, but it requires a bunch of work since you have to create a reproduction repo. This is easy to understand, and should be in a unit test somewhere.

[austin-raey]

I'd like to concur on this issue,

The codebase I'm working with is quite complex so there is likely a lot of moving parts causing this discrepancy, however I would like to note:

I have a pretty simple layout file that is shared for a few pages, seen with this code:

// layout-component.tsx
import Script from 'next/script';

import Footer from './footer';
import Header from './header';
import Main from './main';

export default function DefaultLayout({
  children,
}: Readonly<{
  children?: React.ReactNode;
}>) {
  return (
    <>
      {process.env.NODE_ENV === 'production' && <Script src="/analytics.js" />}
      <Header />
      <Main>{children}</Main>
      <Footer />
    </>
  );
}

With that code in that particular file, and a generateBuildId config present to always generate the same Build ID, I will consistently produce page-e1ea5a6b17a0777a.js in .next/static/chunks/app/.... I have replicated this running build many, many times.

However, simply removing the <Script> tag now makes all pages dependent on this have inconsistent filenames in .next/static/chunks/app/..., all else held equal:

// layout-component.tsx, modified to remove <Script />.
import Footer from './footer';
import Header from './header';
import Main from './main';

export default function DefaultLayout({
  children,
}: Readonly<{
  children?: React.ReactNode;
}>) {
  return (
    <>
      <Header />
      <Main>{children}</Main>
      <Footer />
    </>
  );
}

Now, the page.js file will cycle between about 3 or 4 different filenames: page-090c4b84bc1c1c00.js, page-c2924b123f71afcb.js, page-a711e6f388147d72.js. It becomes semi-deterministic?

Completely confounded as to why such a change would cause this issue, especially with a Build ID. Being that we are building to a couple different load balancers, this is a very irritating issue.

I had previously thought this non-deterministic issue was potentially caused by more complex node_module dependencies, but with this I have no idea.

[owenallenaz]

For what it's worth, what we do is docker build which runs the next build and then we re-use the image in two different Kube pods using replicas: 2. So then it's the exact same content, rather than needing to run build twice (which would just be a performance slog in CI anyways).

[austin-raey]

Yep, this is what we're looking at right now. Our operation had gotten by with 2 nodes, so we only needed 2 separate builds. This has generally worked o.k. in the past.

The solution going forward is we're going to just do as you say -- build once, and in our case, export the resulting Docker Image to both nodes. This ensures content is always exactly the same regardless of Build IDs or whatever content hashes come up. I'm sure Kubernetes is geared for this exact use case but I'll have to review this in future 😵‍💫

Honestly, this also seems better practice anyways. 😂 I could imagine scaling to 5 or 10 nodes, and then having to run 5 or 10 separate builds. That doesn't seem optimal lol.

Nonetheless, an interesting issue this is.

[CaptainN]

The problem with this is compile and generate are both done in the build context. In our deployment, we don't even have access to the services that would create the generate artifacts, since build and runtime are in two entirely different bits of infrastructure. But also, you aren't supposed to produce varying build artifacts - that's a docker no-no.

There is a kind of away around it, and it does actually address the bug described in this thread. You can do "compile" in the build environment, which creates all the chunks, and therefor, can't vary the chunk names in the runtime. This is still a bug, and it should still be fixed, because who knows what else it's doing. Then you just have to do generate in the runtime environment at startup. This also speeds up pod startup dramatically.

You still might end up with variable static generation artifacts, but you can work around that by using a custom redis backing store or something.

What would be even better is some straightforward way to just disable static generation at startup of all kinds, and let it fill the ISR cache at runtime. Maybe one day.

[JohnRPB]

This is a bit confounding to me as well. I'm using a consistent generateBuildId() output so my generated webpack files should all have the same names, but a few don't and cause key requests to fail with a 404 error. This means that caching pages across builds doesn't work; you have to delete the page entries in the cache on deployment. Not the behavior I was expecting from generateBuildId() :-/

[CaptainN]

I guess someone is going to have to make a reproduction repro, and file a ticket, even though there's more than enough information on this thread to see the problem...

@JohnRPB That used to work as expected. Now we get variable chunk names. There are workarounds available (described above), but none of that should be necessary.

[michael-meetkai]

hey @CaptainN @JohnRPB have u found a solution ? having the same issue where the server chunks are different between my local env and CI/CD

[CaptainN]

My solution is to split build - compile in GHA, which locks the artifacts, and then do generation in startup. This works because the variable output happens in the compile stage. This relies on side effects being stable, which is not super amazing. It'd be nicer if the next.js team would fix the core issue.

[mhmmdglc]

is there any update ?

[samcarton]

Adding my 2c to this - This is a blocker especially given source maps are either all or nothing. If you want a public build without source maps, and a separate source-mapped build for external monitoring (sentry/datadog etc), the chunk names are going to be different. In this related discussion #7579 (comment) there are only half-solutions which involve stripping out sourcemaps from the deployed build which seems to cause 404s in some browsers.

[mhmmdglc]

I think Vercel is intentionally not solving this so that we use their platform :D