feat(gatsby): Page build optimisations for incremental data changes #21523

dominicfallows · 2020-02-17T13:18:56Z

Description

Gatsby sources data from multiple sources (CMS, static files - like Markdown, databases, APIs, etc) and creates an aggregated dataset in GraphQL. Currently, each gatsby build uses the GraphQL dataset and queries to do a complete rebuild of the whole app - ready for deployment - including static assets like HTML, JavaScript, JSON, media files, etc.

Projects that have a small (10s to 100s) to medium (100s to 1000s) amount of content, deploying these sites don't present a problem.

Building sites with large amounts of content (10,000s upwards) are already relatively fast with Gatsby. However, some projects might start to experience issues when adopting CI/CD principles - continuously building and deploying. Gatsby rebuilds the complete app which means the complete app also needs to be deployed. Doing this each time a small data change occurs unnecessarily increases demand on CPU, memory, and bandwidth.

One solution to these problems might be to use Gatsby Cloud's Build features.

For projects that require self-hosted environments, where Gatsby Cloud would not be an option, being able to only deploy the content that has changed or is new (incremental data changes, you might say) would help reduce build times, deployment times and demand on resources.

This PR is to introduce an experimental enhancement to only build pages with data changes.

How to use

To enable this enhancement, use the environment variable GATSBY_PAGE_BUILD_ON_DATA_CHANGES=true in your gatsby build command, for example:

GATSBY_PAGE_BUILD_ON_DATA_CHANGES=true node ./node_modules/.bin/gatsby build

This will run the Gatsby build process, but only build pages that have data changes since your last build. If there are any changes to code (JS, CSS) the bundling process returns a new webpack compilation hash which causes all pages to be rebuilt.

Reporting what has been built

You might need to get a list of the pages that have been built for example, if you want to perform a sync action in your CI/CD pipeline.

To list the paths in the build assets (public) folder, you can use one (or both) of the following arguments in your build command.

--log-pages outputs the updated paths to the console at the end of the build

success Building production JavaScript and CSS bundles - 82.198s
success run queries - 82.762s - 4/4 0.05/s
success Building static HTML for pages - 19.386s - 2/2 0.10/s
+ success Delete previous page data - 1.512s
info Done building in 152.084 sec
+ info Built pages:
+ Updated page: /about
+ Updated page: /accounts/example
+ info Deleted pages:
+ Deleted page: /test

Done in 154.501 sec

--write-to-file creates two files in the .cache folder, with lists of the changes paths in the build assets (public) folder.
- newPages.txt will contain a list of paths that have changed or are new
- deletedPages.txt will contain a list of paths that have been deleted

If there are no changed or deleted paths, then the relevant files will not be created in the .cache folder.

Approach

An enhancement works by comparing the previous page data from cache (returned by readState()) to the newly created page data in GraphQL, that can be accessed by store.getState(). By comparing these two data sets, we can determine which pages have been updated, newly created or deleted.

There are two new functions getChangedPageDataKeys and removePreviousPageData in utils/page-data.js:

getChangedPageDataKeys loops through each page's "content" this includes the data and context, comparing it to the previous content. If there is a difference, or the key does not exist (new page), this key is added to this functions returned array.
removePreviousPageData loops through each key, if the key is not present in the new data, the page will be removed and a key added to this functions returned array.

This array of path keys used as the pagePaths value for the buildHTML.buildPages process.

At the end of the build process, the removePreviousPageData function uses each deleted page key to remove a matching directory from the public folder. This is instead of deleting all HTML from the public directory at the beginning of the build process.

Performance improvement

We have run various performance tests on our projects. For context, we use AWS CodePipeline to build and deploy our Gatsby projects, one of which is approaching 30k pages.

On our ~30k page project, when we run a full build versus a content change build, we are seeing vastly improved deploy times, alongside reduced CPU and memory spikes.

For example, for a full build and deploy, we see an average of 10-11 minutes. For a content change build, this is reduced down to an average 5-6 minutes 🚀

Further considerations

To enable this build option you will need to set an environment variable, so you will need access to set variables in your build environment.
You will need to persist the.cache/redux.state between builds, allowing for comparison, if there is no redux.state file located in the /.cache the folder then a full build will be triggered.
Any code or static query changes (templates, components, source handling, new plugins etc) creates a new webpack compilation hash and triggers a full build.

Related Issues

Related to PR #20785
Related to Issue #5002

…ctive-investor/gatsby into improve-page-build-on-data-change

freiksenet

Amazing work 👍

Some comments.

freiksenet · 2020-02-17T13:33:09Z

packages/gatsby/src/commands/build.js

-
-  const pagePaths = [...store.getState().pages.keys()]
+  const pagePaths = process.env.GATSBY_PAGE_BUILD_ON_DATA_CHANGES
+    ? await pageDataUtil.getChangedPageDataKeys(store.getState(), readState())


We already read the state when we initialize, maybe we can save reference to required old state (webpack hash, pages and pageData). I think if we do at the beginning of build (before bootstrap), it should have old page data. I'm kinda wary about re-reading cache multiple times, who knows when it can get changed.

Yep that makes sense 👍

Ok that's done 🙇

freiksenet · 2020-02-17T13:33:39Z

packages/gatsby/src/commands/build.js

+    deletedPageKeys = await pageDataUtil.removePreviousPageData(
+      program.directory,
+      store.getState(),
+      readState()


Ditto about not re-reading the cache here.

freiksenet · 2020-02-17T13:35:22Z

packages/gatsby/src/query/index.js

+    const { pageData } = readState()
+
+    if (pageData) {
+      readState().pageData.forEach((value, key) => {


Maybe we can pass the cached pageData from the build function here.

Can eliminate the explicit if branch by:

Suggested change

readState().pageData.forEach((value, key) => {

pageData?.forEach((value, key) => {

I'm getting a no-unused-expressions lint error 🤔

I would argue to keep original code here (maybe just lose readState().pageData to pageData - the optional chaining here makes it more difficult to read what's going on ;)

@StuartRayson That's because we're currently using an old version of ESLint that doesn't understand optional chaining

packages/gatsby/src/utils/page-data.js

sidharthachatterjee · 2020-02-17T13:47:33Z

Excellent work!

--log-pages outputs the updated paths to the console at the end of the build
success Building production JavaScript and CSS bundles - 82.198s
success run queries - 82.762s - 4/4 0.05/s
success Building static HTML for pages - 19.386s - 2/2 0.10/s

success Delete previous page data - 1.512s
info Done building in 152.084 sec

info Built pages:

Updated page: /about

Updated page: /accounts/example

info Deleted pages:

Deleted page: /test

Done in 154.501 sec
--write-to-file creates two files in the .cache folder, with lists of the changes paths in the build assets (public) folder.

newPages.txt will contain a list of paths that have changed or are new
deletedPages.txt will contain a list of paths that have been deleted
If there are no changed or deleted paths, then the relevant files will not be created in the .cache folder.

Can we add some documentation for this? Cc @ascorbic We'll want to standardise on this

packages/gatsby/src/query/queue.js

pvdz

Thank you for this PR :) I mostly reviewed superficial syntactical comments that stuck out to me.

packages/gatsby/src/commands/build.js

pvdz · 2020-02-17T14:33:04Z

packages/gatsby/src/commands/build.js

+        )}`.replace(/,/g, ``)
+      )
+    }
+    if (typeof deletedPageKeys !== `undefined` && deletedPageKeys.length) {


Assuming null is not an important distinction, you can simplify this;

Suggested change

if (typeof deletedPageKeys !== `undefined` && deletedPageKeys.length) {

if (deletedPageKeys?.length) {

packages/gatsby/src/commands/build.js

pvdz · 2020-02-17T14:36:54Z

packages/gatsby/src/query/index.js

+    const { pageData } = readState()
+
+    if (pageData) {
+      readState().pageData.forEach((value, key) => {


Can eliminate the explicit if branch by:

Suggested change

readState().pageData.forEach((value, key) => {

pageData?.forEach((value, key) => {

packages/gatsby/src/utils/page-data.js

packages/gatsby/src/query/index.js

pieh · 2020-02-28T11:54:25Z

Another gatsby@pr-21523 canary released (containing latest changes).

From code perspective this is ready to be merged (IMO)

laurieontech

From a documentation standpoint, this is very clear and well written. Well done!
I've made a few suggestions, particularly to the important last point. But other than that the text should be ready.

docs/docs/page-build-optimizations-for-incremental-data-changes.md

Co-Authored-By: LB <laurie@gatsbyjs.com>

pieh · 2020-02-28T17:27:56Z

Ops, changes from suggestions to docs broke linting. Please run yarn format:other in root of repo (or allow us to push changes to this branch - https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork )

docs/docs/page-build-optimizations-for-incremental-data-changes.md

muescha · 2020-02-28T22:35:18Z

packages/gatsby/src/utils/worker/render-html.js

+            resolve(
+              fs.outputFile(
+                getPageHtmlFilePath(join(process.cwd(), `public`), path),
+                htmlString
+              )


just interested :

everywhere there is publicDir as param, but here is join(process.cwd(), `public`) used with process.cwd()

i think all the publicDir comes from the following snippet and use program.directory?

https://github.com/interactive-investor/gatsby/blob/0be16c63cc4da72f9d1049c8aa6d737926d62dfd/packages/gatsby/src/commands/build.js#L49-L50

module.exports = async function build(program: BuildArgs) { const publicDir = path.join(program.directory, `public`)

Where renderHTML is used in commands/build-html.ts, doBuildPages is not passed program as a parameter.

gatsby/packages/gatsby/src/commands/build-html.ts

Line 137 in 7527993

await doBuildPages(rendererPath, pagePaths, activity, workerPool)

maybe this is to avoid program being passed unnecessarily to other functions.

We actually should be passing program around (or at least build directory) instead of using process.cwd(), but this doesn't change anything in practice for now (as process.cwd() was already used before).

We do want to make it so we don't use process.cwd() at all, as this make it hard for us to enable options like changing build directory in future, or even things like triggering builds programmatically (instead of using CLI), but this is not concern of this PR, so it's ok for it to stay like that for now

I have noted this in mind of future change of output dir

Yes - it is not the scope of this PR

freiksenet

LGTM. Great job! 👍

done

StuartRayson · 2020-03-02T15:28:03Z

@pieh are you ok to review this?

pieh · 2020-03-02T15:35:39Z

@pieh are you ok a review this?

From my perspective this is good to go and I have no further comments/reviews.

I do want to give some time for @sidharthachatterjee and/or @ascorbic to give either go-ahead or leave comments

ascorbic · 2020-03-02T17:13:38Z

We'll be looking at this tomorrow.

sidharthachatterjee

Looks great! Let's get this in 🎉

sidharthachatterjee · 2020-03-02T08:06:43Z

packages/gatsby/src/utils/page-html.js

@@ -0,0 +1,26 @@
+const fs = require(`fs-extra`)


Should probably move these to gatsby-core-utils later so they are reusable across gatsby packages

pieh

Let's get this in!

Thank you @dominicfallows and @StuartRayson for work put into this feature!

pieh · 2020-03-03T10:52:55Z

Published gatsby@2.19.27 with support for experimental GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES env var!

dominicfallows · 2020-03-03T11:46:52Z

Brill, thanks all for the collab!

StuartRayson · 2020-03-03T12:28:30Z

🙌 I've really enjoyed working on this ! 🚀

lqze · 2020-03-13T04:15:33Z

Has anybody tested this with gatsby-source-wordpress? I am testing locally to see any changes with the log file output, however it seems all of my pages are being updated, even when just running the command right after a successful run, without any changes to content/data.

sharanrprasad · 2020-04-02T22:36:30Z

Hello everyone. Great work! I was trying this out on one of our existing CI/CD pipelines. When ever a page is removed from our CMS incremental build fails. Should all the contents of the public folder be there for the build to pass.
Our site is hosted on S3. Currently I am persisting the cache between each builds using another S3 bucket. If all the contents of pubic folder from previous build also needs to be present then I have to pull content for about 10000 pages and that adds about 2-3 minutes to the build time. Is there a way that I can only pull some of the existing public folder contents or nothing at all but the cache?

StuartRayson · 2020-04-16T08:08:37Z

@sharanrprasad
Yep the public folder needs to be persisted, removing the folder can cause issues when running static queries. We had a similar issues storing and moving our built assets on S3 buckets, so we moved to use EFS and mounting.

aheissenberger · 2020-05-06T18:58:43Z

Problem:
rerunning the code build without any change in the repository will always trigger a full build

GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --write-to-file --log-pages

Environment:

AWS Codebuild / aws/codebuild/standard:4.0 / node 12.16.1
Gatsby 2.21.16

.cache and public are saved in s3 bucket by AWS CodeBuild:

cache:
   paths:
      - '.cache/**/*'
      - 'public/**/*'

Running the the same command on my local environment will do an incremental build.

Are there any flags which will allow to debug the reason why the cache used is not valid?

wardpeet · 2020-05-07T08:17:50Z

Sadly, no. Maybe you're using something along the lines of new Date() Best is to check global queries

StuartRayson and others added 11 commits February 5, 2020 10:05

store page data between builds

8937307

Support removing of pages from public

13b4071

build all pages if webpackCompilationHash has changed

6736205

Support deleting of data on processors that run once

bfd281e

Add expirement flag to page performance

30e041d

Add expirement flag to page performance

ad83b98

Add comments and types to actions

a47aa30

Add missing page data reducer

d9aaac4

Retain public between builds

635bbe4

Add docs to page build time enhancement

3555004

Update Page build optimisations docs

326d412

dominicfallows added the type: feature or enhancement label Feb 17, 2020

dominicfallows requested review from freiksenet and wardpeet February 17, 2020 13:18

dominicfallows requested review from a team as code owners February 17, 2020 13:18

This was referenced Feb 17, 2020

question - incremental builds support = part II #5002

Closed

Explore page build optimisations #20785

Closed

StuartRayson added 2 commits February 17, 2020 13:23

Fix build.js conflict

584ca37

Merge branch 'improve-page-build-on-data-change' of github.com:intera…

3d595a4

…ctive-investor/gatsby into improve-page-build-on-data-change

freiksenet suggested changes Feb 17, 2020

View reviewed changes

pieh reviewed Feb 17, 2020

View reviewed changes

packages/gatsby/src/query/queue.js Outdated Show resolved Hide resolved

pvdz reviewed Feb 17, 2020

View reviewed changes

pieh reviewed Feb 17, 2020

View reviewed changes

packages/gatsby/src/query/index.js Outdated Show resolved Hide resolved

StuartRayson added 4 commits February 17, 2020 15:52

initial refactor from code review

e9b8492

Use hash instead of whole page context

d4d8a67

Use hash instead of whole page context

4af1936

Remove page data in dev action deleteComponentsDependencies

ca51945

handle .html paths when removing empty directories

fef05f5

laurieontech reviewed Feb 28, 2020

View reviewed changes

Apply suggestions from code review

a31749c

Co-Authored-By: LB <laurie@gatsbyjs.com>

muescha reviewed Feb 28, 2020

View reviewed changes

docs/docs/page-build-optimizations-for-incremental-data-changes.md Outdated Show resolved Hide resolved

muescha reviewed Feb 28, 2020

View reviewed changes

fix formating issue on docs

09cc305

freiksenet reviewed Mar 2, 2020

View reviewed changes

freiksenet changed the title ~~Page build optimisations for incremental data changes~~ feat(gatsby): Page build optimisations for incremental data changes Mar 2, 2020

sidharthachatterjee self-requested a review March 3, 2020 10:39

sidharthachatterjee approved these changes Mar 3, 2020

View reviewed changes

pieh approved these changes Mar 3, 2020

View reviewed changes

pieh merged commit c49d7b4 into gatsbyjs:master Mar 3, 2020

pieh added a commit to pieh/gatsby that referenced this pull request Mar 3, 2020

test: fix snapshot after merging gatsbyjs#21523

4bb06ea

pieh mentioned this pull request Mar 3, 2020

test: fix snapshot after merging #21523 #21931

Merged

gatsbybot pushed a commit that referenced this pull request Mar 3, 2020

test: fix snapshot after merging #21523 (#21931)

94cb77a

karlhorky mentioned this pull request Sep 28, 2020

Feature Request: Skip / Speedup Image Generation Processes #24822

Closed

	readState().pageData.forEach((value, key) => {
	pageData?.forEach((value, key) => {

	if (typeof deletedPageKeys !== `undefined` && deletedPageKeys.length) {
	if (deletedPageKeys?.length) {

feat(gatsby): Page build optimisations for incremental data changes #21523

feat(gatsby): Page build optimisations for incremental data changes #21523

Conversation

dominicfallows commented Feb 17, 2020

Description

How to use

Reporting what has been built

Approach

Performance improvement

Further considerations

Related Issues

freiksenet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sidharthachatterjee commented Feb 17, 2020 • edited Loading

pvdz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pieh commented Feb 28, 2020

laurieontech left a comment

Choose a reason for hiding this comment

pieh commented Feb 28, 2020

muescha Feb 28, 2020 • edited Loading

Choose a reason for hiding this comment

StuartRayson Feb 29, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freiksenet left a comment

Choose a reason for hiding this comment

StuartRayson commented Mar 2, 2020 • edited Loading

pieh commented Mar 2, 2020

ascorbic commented Mar 2, 2020

sidharthachatterjee left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pieh left a comment

Choose a reason for hiding this comment

pieh commented Mar 3, 2020

dominicfallows commented Mar 3, 2020

StuartRayson commented Mar 3, 2020

lqze commented Mar 13, 2020

sharanrprasad commented Apr 2, 2020 • edited Loading

StuartRayson commented Apr 16, 2020 • edited Loading

aheissenberger commented May 6, 2020

wardpeet commented May 7, 2020

sidharthachatterjee commented Feb 17, 2020 •

edited

Loading

muescha Feb 28, 2020 •

edited

Loading

StuartRayson Feb 29, 2020 •

edited

Loading

StuartRayson commented Mar 2, 2020 •

edited

Loading

sharanrprasad commented Apr 2, 2020 •

edited

Loading

StuartRayson commented Apr 16, 2020 •

edited

Loading