Next.js: Fix webpack fsCache not working #29654

sentience · 2024-11-19T07:19:37Z

What I did

Next.js overrides certain internal webpack packages (particularly webpack-sources), which are involved with filesystem caching, with its own versions from next/dist/compiled.

For filesystem caching to work, Next.js must be allowed to perform these overrides before webpack is first initialized by @storybook/builder-webpack5. If it is not, the objects to be serialized to disk in the caching process will be instantiated using the original (non-Next.js) modules, but the cache serializers will be created using the Next.js modules. This mismatch between the objects to be cached and the serializers that write the filesystem cache prevents the cache from being written; instead, webpack outputs a warning message to the console for every object that it tries and fails to find a matching serializer for.

This fix works by invoking Next.js to configure webpack in the core hook of @storybook/nextjs/preset, immediately before loading @storybook/builder-webpack5. We don't actually use this configuration that Next.js creates; the actual configuration that will be used in the build is still generated in webpackFinal as before.

fsCache (docs) has a large impact on Storybook build performance. Even in a minimal project with a single story, enabling it reduces hot-cache build time by 66%! This is therefore a very valuable option to be able to enable.

Checklist for Contributors

Testing

@storybook/nextjs does not appear to have a test suite, so I don't see an easy way to test this feature. I've verified that it resolves the issue in my repro project for #29621.

The changes in this PR are covered in the following automated tests:

stories
unit tests
integration tests
end-to-end tests

Manual testing

Clone https://github.com/sentience/storybook-nextjs-webpack-cache-issue-demo
Run Storybook (pnpm i, pnpm storybook build) and note the console warnings at the end of the build that indicate the filesystem cache is not working.
Upgrade @storybook/nextjs to a canary release with this change.
Run Storybook again and note the console warnings are gone, and the build runs in 33% of the time!

Documentation

~~Add or update documentation reflecting your changes~~
If you are deprecating/removing a feature, make sure to update
MIGRATION.MD

Checklist for Maintainers

When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli-storybook/src/sandbox-templates.ts
Make sure this PR contains one of the labels below:
Available labels
- bug: Internal changes that fixes incorrect behavior.
- maintenance: User-facing maintenance tasks.
- dependencies: Upgrading (sometimes downgrading) dependencies.
- build: Internal-facing build tooling & test updates. Will not show up in release changelog.
- cleanup: Minor cleanup style change. Will not show up in release changelog.
- documentation: Documentation only changes. Will not show up in release changelog.
- feature request: Introducing a new feature.
- BREAKING CHANGE: Changes that break compatibility in some way with current major version.
- other: Changes that don't fit in the above categories.

🦋 Canary release

This pull request has been released as version 0.0.0-pr-29654-sha-83a1f24f. Try it out in a new sandbox by running npx storybook@0.0.0-pr-29654-sha-83a1f24f sandbox or in an existing project with npx storybook@0.0.0-pr-29654-sha-83a1f24f upgrade.

More information


Published version	`0.0.0-pr-29654-sha-83a1f24f`
Triggered by	@valentinpalkovic
Repository	sentience/storybook
Branch	`fix-nextjs-webpack-fscache`
Commit	`83a1f24f`
Datetime	Tue Dec 3 06:19:41 UTC 2024 (`1733206781`)
Workflow run	12134081252

To request a new release of this pull request, mention the @storybookjs/core team.

core team members can create a new canary release here or locally with gh workflow run --repo storybookjs/storybook canary-release-pr.yml --field pr=29654

name	before	after	diff	z	%
createSize	0 B	0 B	0 B	-	-
generateSize	77.7 MB	77.7 MB	0 B	0.65	0%
initSize	133 MB	133 MB	-625 B	0.66	0%
diffSize	55.3 MB	55.3 MB	-625 B	0.66	0%
buildSize	6.87 MB	6.87 MB	41 B	-0.03	0%
buildSbAddonsSize	1.51 MB	1.51 MB	0 B	-0.42	0%
buildSbCommonSize	195 kB	195 kB	0 B	-	0%
buildSbManagerSize	1.86 MB	1.86 MB	0 B	1.52	0%
buildSbPreviewSize	0 B	0 B	0 B	-	-
buildStaticSize	0 B	0 B	0 B	-	-
buildPrebuildSize	3.57 MB	3.57 MB	0 B	-0.05	0%
buildPreviewSize	3.3 MB	3.3 MB	41 B	0.29	0%
testBuildSize	0 B	0 B	0 B	-	-
testBuildSbAddonsSize	0 B	0 B	0 B	-	-
testBuildSbCommonSize	0 B	0 B	0 B	-	-
testBuildSbManagerSize	0 B	0 B	0 B	-	-
testBuildSbPreviewSize	0 B	0 B	0 B	-	-
testBuildStaticSize	0 B	0 B	0 B	-	-
testBuildPrebuildSize	0 B	0 B	0 B	-	-
testBuildPreviewSize	0 B	0 B	0 B	-	-

name	before	after	diff	z	%
createTime	7.5s	6.7s	-772ms	-0.95	-11.5%
generateTime	20.3s	18.2s	-2s -171ms	-0.86	-11.9%
initTime	13.1s	12s	-1s -126ms	-1.01	-9.4%
buildTime	9.5s	7.9s	-1s -596ms	-1.69	🔰-20.1%
testBuildTime	0ms	0ms	0ms	-	-
devPreviewResponsive	5s	4.6s	-312ms	-0.73	-6.7%
devManagerResponsive	3.7s	3.5s	-222ms	-0.69	-6.3%
devManagerHeaderVisible	549ms	539ms	-10ms	-0.5	-1.9%
devManagerIndexVisible	626ms	570ms	-56ms	-0.89	-9.8%
devStoryVisibleUncached	1.8s	1.7s	-112ms	-0.46	-6.5%
devStoryVisible	624ms	625ms	1ms	-0.21	0.2%
devAutodocsVisible	507ms	499ms	-8ms	-0.16	-1.6%
devMDXVisible	483ms	465ms	-18ms	-0.61	-3.9%
buildManagerHeaderVisible	585ms	515ms	-70ms	-0.74	-13.6%
buildManagerIndexVisible	692ms	612ms	-80ms	-0.7	-13.1%
buildStoryVisible	534ms	478ms	-56ms	-0.65	-11.7%
buildAutodocsVisible	410ms	415ms	5ms	-0.5	1.2%
buildMDXVisible	415ms	423ms	8ms	-0.37	1.9%

Greptile Summary

This PR fixes webpack's filesystem caching in @storybook/nextjs by ensuring Next.js can properly override webpack internals before Storybook's webpack builder initialization, significantly improving build performance.

Modified code/frameworks/nextjs/src/preset.ts to load Next.js configuration in the core hook before webpack initialization
Added early webpack configuration setup to allow Next.js to override internal packages like webpack-sources
Enables proper filesystem caching functionality, reducing build times by up to 66%
Maintains existing webpack configuration generation in webpackFinal hook for actual builds

Next.js overrides certain internal webpack packages (particularly `webpack-sources`), which are involved with filesystem caching, with its own versions from next/dist/compiled. For filesystem caching to work, Next.js must be allowed to perform these overrides before webpack is first initialized by @storybook/builder-webpack5. If it is not, the objects to be serialized to disk in the caching process will be instantiated using the original (non-Next.js) modules, but the serializers will be created using the Next.js modules. This mismatch between the objects to be cached and the serializers that write the filesystem cache prevents the cache from being written; instead, webpack outputs a warning message to the console for every object that it tries and fails to find a matching serializer for. This fix works by invoking Next.js to configure webpack in the `core` hook of @storybook/nextjs/preset, immediately before loading @storybook/builder-webpack5. We don't actually use this configuration that Next.js creates; the actual configuration that will be used in the build is still generated in `webpackFinal` as before. `fsCache` has a large impact on Storybook build performance. Even in a minimal project with a single story, enabling it reduces build time by 66%! This is therefore a very valuable option to be able to enable. Fixes storybookjs#29621

greptile-apps

_{1 file(s) reviewed, 1 comment(s)}
_{Edit PR Review Bot Settings | Greptile}

code/frameworks/nextjs/src/preset.ts

nx-cloud · 2024-11-19T07:24:24Z

☁️ Nx Cloud Report

CI is running/has finished running commands for commit 902dc18. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution

🟥 Failed Commands
`nx run-many -t build --parallel=3`

Sent with 💌 from NxCloud.

sentience · 2024-11-25T23:11:28Z

@valentinpalkovic May I request a canary release of this? Sorry if you're not the right person to mention; the PR template says to mention @storybookjs/core, but that doesn't seem to be recognised by GitHub.

valentinpalkovic · 2024-11-26T08:13:30Z

Here we go: https://github.com/storybookjs/storybook/actions/runs/12026402956/job/33525318348

When the job is done, you can see the canary version :)

…ernal instance

sentience · 2024-12-03T01:36:11Z

@valentinpalkovic Thanks to that canary build I found some more places where webpack was being referenced before Next.js replaced it with its own internal instance, causing disk cache serializer mismatch errors. I believe I've now fixed all of these.

Could you roll me another canary build, please? 🙏🏻

valentinpalkovic · 2024-12-03T06:20:00Z

Sure :)
https://github.com/storybookjs/storybook/actions/runs/12134081252

sentience · 2024-12-03T06:57:07Z

I've now tested this in several projects, and I think this is in good enough shape to release. Setting fsCache: true (docs) consistently cuts incremental build times by ~66%.

I am, however, aware of one caveat to this fix: projects that use react-docgen-typescript to parse React components to generate high-quality props metadata.

If you set reactDocgen in your Storybook config like this (docs):

/* .storybook/main.ts */
export default {
  ...
  typescript: {
    reactDocgen: "react-docgen-typescript",
  }
};

…and set fsCache: true, you will still get cache warning messages like this:

<w> [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/Users/kyank/Developer/unified-home/node_modules/.pnpm/@storybook+nextjs@8.4.6_@swc+core@1.7.21_@swc+helpers@0.5.12__@types+webpack@5.28.5_@swc+core_2ylubrpntvrhqftmmdfysdmhxe/node_modules/@storybook/nextjs/dist/swc/next-swc-loader-patch.js??ruleSet[1].rules[18].use!/Users/kyank/Developer/unified-home/apps/unified-home/src/components/HomeTitleBlock/HomeTitleBlock.tsx': No serializer registered for DocGenDependency
<w> while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> Array { 19 items } -> DocGenDependency

These messages complain that DocGenDependency objects cannot be cached, because they do not have a cache serializer registered. I have traced the root cause of this to the @storybook/react-docgen-typescript-plugin package. It includes a pair of imports that reach deeply into webpack:

import makeSerializable from "webpack/lib/util/makeSerializable.js";

import NullDependency from "webpack/lib/dependencies/NullDependency.js";

Next.js overrides a bunch of webpack import paths to redirect consumers to its internal copy of webpack, but these paths are not among them.

As a result, @storybook/react-docgen-typescript-plugin registers its cache serializer for DocGenDependency with the "original" webpack, instead of Next.js's internal webpack, and this causes the "not serializable cache item" warnings.

This should be possible to fix by updating @storybook/react-docgen-typescript-plugin to use the officially exposed API for registering cache serializers, rather than its current approach that depends on internals that are not exposed by Next.js's webpack instance. Looking at mini-css-extract-plugin, for example, it uses the webpack.util.serialization.register API that is officially documented. @storybook/react-docgen-typescript-plugin looks like it would require some refactoring to get the webpack object into scope where needed to use this API, but this should be possible.

In the meantime, however, this PR is no worse than the current behaviour (where all objects print a "not serializable cache item" warning), and even in projects where react-docgen-typescript is used, the partial fix of this PR provides a large boost to incremental build performance.

I therefore propose that this be merged as-is, and I'll open a separate issue to track the warnings in react-docgen-typescript projects.

storybook-pr-benchmarking · 2024-12-03T09:59:44Z

Package Benchmarks

^{Commit: 902dc18, ran on 13 December 2024 at 12:05:16 UTC}

No significant changes detected, all good. 👏

sentience · 2024-12-03T23:37:55Z

Not sure why yarn install failed in CI. Seems to work for me locally.

sentience · 2024-12-09T22:48:45Z

Anything else I can do to help move this along?

valentinpalkovic · 2024-12-10T08:12:40Z

Hi @sentience

Thank you for your contribution. I was on vacation but I am back again. I'll take a close look this week.

greptile-apps bot reviewed Nov 19, 2024

View reviewed changes

code/frameworks/nextjs/src/preset.ts Outdated Show resolved Hide resolved

valentinpalkovic changed the title ~~Fix webpack fsCache not working with @storybook/nextjs~~ Next.js: Fix webpack fsCache not working Nov 19, 2024

valentinpalkovic self-assigned this Nov 19, 2024

valentinpalkovic added bug nextjs labels Nov 19, 2024

sentience added 2 commits November 27, 2024 11:54

Fix crash when framework is not specified with options

0a3e05b

Defer all imports from webpack until after Next.js has loaded its int…

83a1f24

…ernal instance

yannbf added the ci:merged Run the CI jobs that normally run when merged. label Dec 3, 2024

Undo unintentional formatting change

c00c054

sneko mentioned this pull request Dec 4, 2024

[Bug]: ExceedsStorageRange errors + sloooow experience with Storybook 8 + NextJS #26644

Open

valentinpalkovic added 2 commits December 13, 2024 11:09

Merge branch 'next' into fix-nextjs-webpack-fscache

980282f

Fix webpack DefinePlugin import to use default export

902dc18

valentinpalkovic merged commit cf24c98 into storybookjs:next Dec 13, 2024
61 of 65 checks passed

github-actions bot mentioned this pull request Dec 13, 2024

Release: Prerelease 8.5.0-alpha.22 #30041

Merged

17 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Next.js: Fix webpack fsCache not working #29654

Next.js: Fix webpack fsCache not working #29654

sentience commented Nov 19, 2024 •

edited by storybook-pr-benchmarking bot

Loading

greptile-apps bot left a comment

nx-cloud bot commented Nov 19, 2024 •

edited

Loading

sentience commented Nov 25, 2024 •

edited

Loading

valentinpalkovic commented Nov 26, 2024

sentience commented Dec 3, 2024

valentinpalkovic commented Dec 3, 2024

sentience commented Dec 3, 2024 •

edited

Loading

storybook-pr-benchmarking bot commented Dec 3, 2024 •

edited

Loading

sentience commented Dec 3, 2024

sentience commented Dec 9, 2024

valentinpalkovic commented Dec 10, 2024

Next.js: Fix webpack fsCache not working #29654

Next.js: Fix webpack fsCache not working #29654

Conversation

sentience commented Nov 19, 2024 • edited by storybook-pr-benchmarking bot Loading

What I did

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

Manual testing

Documentation

Checklist for Maintainers

🦋 Canary release

Greptile Summary

greptile-apps bot left a comment

Choose a reason for hiding this comment

nx-cloud bot commented Nov 19, 2024 • edited Loading

☁️ Nx Cloud Report

sentience commented Nov 25, 2024 • edited Loading

valentinpalkovic commented Nov 26, 2024

sentience commented Dec 3, 2024

valentinpalkovic commented Dec 3, 2024

sentience commented Dec 3, 2024 • edited Loading

storybook-pr-benchmarking bot commented Dec 3, 2024 • edited Loading

Package Benchmarks

sentience commented Dec 3, 2024

sentience commented Dec 9, 2024

valentinpalkovic commented Dec 10, 2024

sentience commented Nov 19, 2024 •

edited by storybook-pr-benchmarking bot

Loading

nx-cloud bot commented Nov 19, 2024 •

edited

Loading

sentience commented Nov 25, 2024 •

edited

Loading

sentience commented Dec 3, 2024 •

edited

Loading

storybook-pr-benchmarking bot commented Dec 3, 2024 •

edited

Loading