feat(gatsby): Page build optimisations for incremental data changes

docs/docs/overview-of-the-gatsby-build-process.md

@@ -301,6 +301,8 @@ Page queries that were queued up earlier from query extraction are run so the da
 
 With everything ready for the HTML pages in place, HTML is compiled and written out to files so it can be served up statically. Since HTML is being produced in a Node.js server context, [references to browser APIs like `window` can break the build](/docs/debugging-html-builds/) and must be conditionally applied.
 
+By default, Gatsby rebuilds static HTML for all pages on each build. There is an experimental feature flag `GATSBY_PAGE_BUILD_ON_DATA_CHANGES` which enables [Page Build Optimizations for Incremental Data Changes](/docs/page-build-optimizations-for-incremental-data-changes/).
+
 ## What do you get from a successful build?
 
 When a Gatsby build is successfully completed, everything you need to deploy your site ends up in the `public` folder at the root of the site. The build includes minified files, transformed images, JSON files with information and data for each page, static HTML for each page, and more.

docs/docs/page-build-optimizations-for-incremental-data-changes.md

@@ -0,0 +1,80 @@
+---
+title: Page Build Optimizations for Incremental Data Changes
+---
+
+Gatsby sources data from multiple sources (CMS, static files - like Markdown, databases, APIs, etc) and creates an aggregated dataset in GraphQL. Currently, each Gatsby build uses the GraphQL dataset and queries to do a complete rebuild of the whole app - ready for deployment - including static assets like HTML, JavaScript, JSON, media files, etc.
+
+For projects that have a small (10s to 100s) to medium (100s to 1000s) amount of content, deployment is not a challenge.
+
+Even building sites with large amounts of content (10,000s upwards) are relatively fast with Gatsby. However, some projects might start to experience issues when adopting CI/CD principles - continuously building and deploying. Gatsby rebuilds the complete app which means the complete app also needs to be deployed. Doing this each time a small data change occurs unnecessarily increases demand on CPU, memory, and bandwidth.
+
+One solution to these problems might be to use [Gatsby Cloud's Build features](https://www.gatsbyjs.com/cloud/).
+
+For projects that require self-hosted environments, where Gatsby Cloud would not be an option, being able to only deploy the content that has changed or is new (incremental data changes, you might say) would help reduce build times, deployment times and demand on resources.
+
+For more info on the standard build process please see [overview of the gatsby build process](/docs/overview-of-the-gatsby-build-process/)
+
+## How to use
+
+To enable this enhancement, use the environment variable `GATSBY_PAGE_BUILD_ON_DATA_CHANGES=true` in your `gatsby build` command, for example:
+
+`GATSBY_PAGE_BUILD_ON_DATA_CHANGES=true node ./node_modules/.bin/gatsby build`
+
+This will run the Gatsby build process, but only build pages that have data changes since your last build. If there are any changes to code (JS, CSS) the bundling process returns a new webpack compilation hash which causes all pages to be rebuilt.
+
+### Reporting what has been built
+
+You may want to retrieve a list of the pages that were built. For example, if you want to perform a sync action in your CI/CD pipeline.
+
+To list the paths in the build assets (`public`) folder, you can use one (or both) of the following arguments in your `build` command.
+
+- `--log-pages` outputs the updated paths to the console at the end of the build
+
+```bash
+success Building production JavaScript and CSS bundles - 82.198s
+success run queries - 82.762s - 4/4 0.05/s
+success Building static HTML for pages - 19.386s - 2/2 0.10/s
++ success Delete previous page data - 1.512s
+info Done building in 152.084 sec
++ info Built pages:
++ Updated page: /about
++ Updated page: /accounts/example
++ info Deleted pages:
++ Deleted page: /test
+
+Done in 154.501 sec
+```
+
+- `--write-to-file` creates two files in the `.cache` folder, with lists of the changes paths in the build assets (`public`) folder.
+
+  - `newPages.txt` will contain a list of paths that have changed or are new
+  - `deletedPages.txt` will contain a list of paths that have been deleted
+
+If there are no changed or deleted paths, then the relevant files will not be created in the `.cache` folder.
+
+The `--log-pages` parameter will output all the file paths that were updated or deleted at the end of the build stage.
+
+```bash
+success Building production JavaScript and CSS bundles - 82.198s
+success run queries - 82.762s - 4/4 0.05/s
+success Building static HTML for pages - 19.386s - 2/2 0.10/s
++ success Delete previous page data - 1.512s
+info Done building in 152.084 sec
++ info Built pages:
++ Updated page: /about
++ Updated page: /accounts/example
++ info Deleted pages:
++ Deleted page: /test
+
+Done in 154.501 sec
+```
+
+## More information
+
+- This enhancement works by comparing the page data from the previous build to the new page data. This creates a list of page directories that are passed to the static build process
+
+- To enable this build option you will need to set an environment variable, so you will need access to set variables in your build environment
+
+- At the end of each build, gatsby creates a `redux.state` file in `/.cache` that contains the all previous build data. You will need to persist the `.cache/redux.state` between builds, allowing for comparison, if there is no `redux.state` file located in the `/.cache` folder then a full build will be triggered
+
+- Any code or static query changes (templates, components, source handling, new plugins etc) creates a new webpack compilation hash and triggers a full build

packages/gatsby/src/bootstrap/index.js

@@ -190,7 +190,10 @@ module.exports = async (args: BootstrapArgs) => {
 
   // During builds, delete html and css files from the public directory as we don't want
   // deleted pages and styles from previous builds to stick around.
-  if (process.env.NODE_ENV === `production`) {
+  if (
+    !process.env.GATSBY_PAGE_BUILD_ON_DATA_CHANGES &&
+    process.env.NODE_ENV === `production`
+  ) {
     activity = report.activityTimer(
       `delete html and css files from previous builds`,
       {

packages/gatsby/src/bootstrap/page-hot-reloader.js

@@ -1,7 +1,11 @@
 const { emitter, store } = require(`../redux`)
 const apiRunnerNode = require(`../utils/api-runner-node`)
 const { boundActionCreators } = require(`../redux/actions`)
-const { deletePage, deleteComponentsDependencies } = boundActionCreators
+const {
+  deletePage,
+  deleteComponentsDependencies,
+  removePageData,
+} = boundActionCreators
 const report = require(`gatsby-cli/lib/reporter`)
 
 let pagesDirty = false
@@ -34,6 +38,7 @@ const runCreatePages = async () => {
       page.path !== `/404.html`
     ) {
       deleteComponentsDependencies([page.path])
+      removePageData(page.path)
       deletePage(page)
     }
   })

packages/gatsby/src/commands/build.js

@@ -2,6 +2,7 @@
 
 const path = require(`path`)
 const report = require(`gatsby-cli/lib/reporter`)
+const fs = require(`fs-extra`)
 import { buildHTML } from "./build-html"
 const buildProductionBundle = require(`./build-javascript`)
 const bootstrap = require(`../bootstrap`)
@@ -11,14 +12,17 @@ const { initTracer, stopTracer } = require(`../utils/tracer`)
 const db = require(`../db`)
 const signalExit = require(`signal-exit`)
 const telemetry = require(`gatsby-telemetry`)
-const { store, emitter } = require(`../redux`)
+const { store, emitter, readState } = require(`../redux`)
 const queryUtil = require(`../query`)
 const appDataUtil = require(`../utils/app-data`)
 const WorkerPool = require(`../utils/worker/pool`)
 const { structureWebpackErrors } = require(`../utils/webpack-error-utils`)
 const {
   waitUntilAllJobsComplete: waitUntilAllJobsV2Complete,
 } = require(`../utils/jobs-manager`)
+const pageDataUtil = require(`../utils/page-data`)
+
+const cacheData = readState()
 
 type BuildArgs = {
   directory: string,
@@ -70,6 +74,7 @@ module.exports = async function build(program: BuildArgs) {
     processStaticQueries,
   } = queryUtil.getInitialQueryProcessors({
     parentSpan: buildSpan,
+    cacheData,
   })
 
   await processStaticQueries()
@@ -137,14 +142,11 @@ module.exports = async function build(program: BuildArgs) {
     `BOOTSTRAP_QUERY_RUNNING_FINISHED`
   )
 
-  await db.saveState()
-
   await waitUntilAllJobsComplete()
 
-  // we need to save it again to make sure our latest state has been saved
-  await db.saveState()
-
-  const pagePaths = [...store.getState().pages.keys()]
+  const pagePaths = process.env.GATSBY_PAGE_BUILD_ON_DATA_CHANGES
+    ? pageDataUtil.getChangedPageDataKeys(store.getState(), cacheData)
+    : [...store.getState().pages.keys()]
   activity = report.createProgress(
     `Building static HTML for pages`,
     pagePaths.length,
@@ -184,6 +186,28 @@ module.exports = async function build(program: BuildArgs) {
   }
   activity.done()
 
+  let deletedPageKeys = []
+  if (process.env.GATSBY_PAGE_BUILD_ON_DATA_CHANGES) {
+    activity = report.activityTimer(`Delete previous page data`)
+    activity.start()
+    deletedPageKeys = pageDataUtil.removePreviousPageData(
+      store.getState(),
+      cacheData
+    )
+
+    deletedPageKeys.forEach(value => {
+      if (value === `/`) {
+        fs.removeSync(`${program.directory}/public/index.html`)
+        fs.removeSync(`${program.directory}/public/page-data/index`)
+      } else {
+        fs.removeSync(`${program.directory}/public${value}`)
+        fs.removeSync(`${program.directory}/public/page-data${value}`)
+      }
+    })
+
+    activity.end()
+  }
+
   activity = report.activityTimer(`onPostBuild`, { parentSpan: buildSpan })
   activity.start()
   await apiRunnerNode(`onPostBuild`, {
@@ -201,4 +225,52 @@ module.exports = async function build(program: BuildArgs) {
   await stopTracer()
   workerPool.end()
   buildActivity.end()
+
+  if (
+    process.env.GATSBY_PAGE_BUILD_ON_DATA_CHANGES &&
+    process.argv.includes(`--log-pages`)
+  ) {
+    if (pagePaths.length) {
+      report.info(
+        `Built pages:\n${pagePaths
+          .map(path => `Updated page: ${path}`)
+          .join(`\n`)}`
+      )
+    }
+
+    if (deletedPageKeys.length) {
+      report.info(
+        `Deleted pages:\n${deletedPageKeys
+          .map(path => `Deleted page: ${path}`)
+          .join(`\n`)}`
+      )
+    }
+  }
+
+  if (
+    process.env.GATSBY_PAGE_BUILD_ON_DATA_CHANGES &&
+    process.argv.includes(`--write-to-file`)
+  ) {
+    const createdFilesPath = path.resolve(
+      `${program.directory}/.cache`,
+      `newPages.txt`
+    )
+    const deletedFilesPath = path.resolve(
+      `${program.directory}/.cache`,
+      `deletedPages.txt`
+    )
+
+    if (pagePaths.length) {
+      fs.writeFileSync(createdFilesPath, `${pagePaths.join(`\n`)}\n`, `utf8`)
+      report.info(`newPages.txt created`)
+    }
+    if (deletedPageKeys.length) {
+      fs.writeFileSync(
+        deletedFilesPath,
+        `${deletedPageKeys.join(`\n`)}\n`,
+        `utf8`
+      )
+      report.info(`deletedPages.txt created`)
+    }
+  }
 }

packages/gatsby/src/query/index.js

@@ -158,9 +158,21 @@ const groupQueryIds = queryIds => {
   }
 }
 
-const processQueries = async (queryJobs, activity) => {
+const processQueries = async (queryJobs, activity, cacheData) => {
   const queue = queryQueue.createBuildQueue()
   await queryQueue.processBatch(queue, queryJobs, activity)
+  if (process.env.GATSBY_PAGE_BUILD_ON_DATA_CHANGES) {
+    const { pages } = store.getState()
+    if (cacheData && cacheData.pageData) {
+      cacheData.pageData.forEach((_value, key) => {
+        if (!pages.has(key)) {
+          boundActionCreators.removePageData({
+            id: key,
+          })
+        }
+      })
+    }
+  }
 }
 
 const createStaticQueryJob = (state, queryId) => {
@@ -206,7 +218,7 @@ const processStaticQueries = async (queryIds, { state, activity }) => {
   )
 }
 
-const processPageQueries = async (queryIds, { state, activity }) => {
+const processPageQueries = async (queryIds, { state, activity }, cacheData) => {
   state = state || store.getState()
   // Make sure we filter out pages that don't exist. An example is
   // /dev-404-page/, whose SitePage node is created via
@@ -215,11 +227,12 @@ const processPageQueries = async (queryIds, { state, activity }) => {
   const pages = _.filter(queryIds.map(id => state.pages.get(id)))
   await processQueries(
     pages.map(page => createPageQueryJob(state, page)),
-    activity
+    activity,
+    cacheData
   )
 }
 
-const getInitialQueryProcessors = ({ parentSpan } = {}) => {
+const getInitialQueryProcessors = ({ parentSpan, cacheData } = {}) => {
   const state = store.getState()
   const queryIds = calcInitialDirtyQueryIds(state)
   const { staticQueryIds, pageQueryIds } = groupQueryIds(queryIds)
@@ -230,12 +243,12 @@ const getInitialQueryProcessors = ({ parentSpan } = {}) => {
 
   let activity = null
   let processedQueuesCount = 0
-  const createProcessor = (fn, queryIds) => async () => {
+  const createProcessor = (fn, queryIds, cacheData) => async () => {
     if (!activity) {
       activity = createQueryRunningActivity(queryjobsCount, parentSpan)
     }
 
-    await fn(queryIds, { state, activity })
+    await fn(queryIds, { state, activity }, cacheData)
 
     processedQueuesCount++
     // if both page and static queries are done, finish activity
@@ -246,7 +259,11 @@ const getInitialQueryProcessors = ({ parentSpan } = {}) => {
 
   return {
     processStaticQueries: createProcessor(processStaticQueries, staticQueryIds),
-    processPageQueries: createProcessor(processPageQueries, pageQueryIds),
+    processPageQueries: createProcessor(
+      processPageQueries,
+      pageQueryIds,
+      cacheData
+    ),
     pageQueryIds,
   }
 }

packages/gatsby/src/query/query-runner.js

@@ -98,7 +98,6 @@ module.exports = async (graphqlRunner, queryJob: QueryJob) => {
     .createHash(`sha1`)
     .update(resultJSON)
     .digest(`base64`)
-
   if (resultHash !== resultHashes.get(queryJob.id)) {
     resultHashes.set(queryJob.id, resultHash)
 
@@ -117,7 +116,6 @@ module.exports = async (graphqlRunner, queryJob: QueryJob) => {
         `d`,
         `${queryJob.hash}.json`
       )
-
       await fs.outputFile(resultPath, resultJSON)
     }
   }
@@ -128,5 +126,12 @@ module.exports = async (graphqlRunner, queryJob: QueryJob) => {
     isPage: queryJob.isPage,
   })
 
+  if (process.env.GATSBY_PAGE_BUILD_ON_DATA_CHANGES && queryJob.isPage) {
+    boundActionCreators.setPageData({
+      id: queryJob.id,
+      result: { resultHash },
+    })
+  }
+
   return result
 }

packages/gatsby/src/query/queue.js

@@ -14,6 +14,7 @@ const createBaseOptions = () => {
 
 const createBuildQueue = () => {
   const graphqlRunner = new GraphQLRunner(store)
+
   const handler = (queryJob, callback) =>
     queryRunner(graphqlRunner, queryJob)
       .then(result => callback(null, result))

packages/gatsby/src/redux/__tests__/__snapshots__/index.js.snap

@@ -20,6 +20,7 @@ Object {
     "complete": Map {},
     "incomplete": Map {},
   },
+  "pageData": Map {},
   "pageDataStats": Map {},
   "staticQueryComponents": Map {},
   "status": Object {