Re-building development bundle slow in v3

[tu4mo]

I updated Gatsby from v2.24.0 to v3.5.0. Re-building development bundle task went from:

success Re-building development bundle - 2.787s

to

success Re-building development bundle - 27.203s

Also, and maybe related, starting the project takes a lot of time:

success Building development bundle - 197.799s

Packages in use:

"gatsby": "^3.5.0",
"gatsby-plugin-react-helmet": "^4.5.0",
"gatsby-plugin-styled-components": "^4.5.0",
"gatsby-source-filesystem": "^3.5.0",
"gatsby-transformer-react-docgen": "^6.5.0",
"gatsby-transformer-remark": "^4.2.0",

Due to private project, I can't provide a reproduction. Are there ways for me to debug the issue?

[LekoArts]

Can you enable the PRESERVE_WEBPACK_CACHE flag? #28331

webpack 5 got on the "cold" build (without cache) slower from 4 => 5. With that flag we activated its cache (and plan to ship it for everyone eventually ofc)

[tu4mo]

I did some testing with a simple webpack project and I'm not seeing this drastic slowdown with webpack 5 re-builds. It seems to me that webpack only re-builds the changed part of my library, but with Gatsby it feels like it re-builds everything (could explain why re-build takes as long as initial build). I also wonder does Gatsby run everything through Babel (possibly even my library)?

[tu4mo]

More findings: When my component library is rebuild, rollup recreates all component files, even unchanged ones. This is probably why Gatsby seems to be re-building everything. If I manually change one compiled file from the library, Gatsby re-builds in under a second. This doesn't seem to happen with my simple webpack project.

[KyleAMathews]

ooo yeah — webpack will always process changed files even if the contents didn't change. So if you can get Rollup not to write out unchanged files, that'd solve it.

[tu4mo]

I don't see rollup doing those kind of changes in the near future. And like I mentioned before, with simple webpack 5 project re-builds are fast.

One thing I just noticed, there was a message in console briefly that was something like: <i> waiting for /common.css, but it was overridden by success Re-building development bundle - 28.225s before I got to copy-paste it. Could the slowness have something to do with that?

[tu4mo]

Also seeing this:

<i> [webpack-dev-middleware] wait until bundle finished: /page-data/app-data.json

 ERROR

Error loading a result for the page query in "sisalto/columnconnector" /
"sisalto/columnconnector". Query was not run and no cached result was found.
Couldn't get query results for "/sisalto/columnconnector/" in 15.000s.



  Error: Couldn't get query results for "/sisalto/columnconnector/" in 15.000s.

  - get-page-data.ts:100
    [styleguide]/[gatsby]/src/utils/get-page-data.ts:100:13

  - start-server.ts:312
    [styleguide]/[gatsby]/src/utils/start-server.ts:312:24

[tu4mo]

Ok, I figured it out. Like I suspected earlier, dependencies installed from workspaces go through babel-loader, maybe because they aren't physically inside node_modules but symlinks. I added my custom exclude for js loaders and re-builds are now just a couple of seconds, which I can live with.

exports.onCreateWebpackConfig = ({
  actions,
  getConfig,
  loaders,
  plugins,
  stage
}) => {
  const currentConfig = getConfig()

  currentConfig.module.rules = [
    ...currentConfig.module.rules.filter(
      rule => String(rule.test) !== String(/\.(js|mjs|jsx)$/)
    ),
    {
      ...loaders.js(),
      test: /\.(js|mjs|jsx)$/,
      exclude: /(node_modules|lib\/components)/
    }
  ]

  actions.replaceWebpackConfig(currentConfig)
}

[tu4mo]

Doing this kind of configuration feels kinda fragile though. Perhaps Gatsby could do something about it?

[KyleAMathews]

It'd be nice to have an "gatsby-plugin-excludes" plugin perhaps to wrap up this pattern. Not sure we'd want to support something more directly.

[tu4mo]

I just published gatsby-plugin-babel-exclude.

[KyleAMathews]

Nice! Could you add the "gatsby-plugin" keyword and republish so it shows up in the plugin library?