perf(gatsby-plugin-contentful): drop the super expensive fixids cycle #27318

pvdz · 2020-10-07T11:39:33Z

This is the diff @axe312ger put up in #25948 but with the last necessary fix.

Also leaving some comments to double check.

This change drops the fixids function call which was applied to the entire model and is very expensive to the Contentful data set. It was also no longer needed so this is a huge win for sites sourcing from Contentful.

packages/gatsby-source-contentful/src/__tests__/__snapshots__/gatsby-node.js.snap

pvdz · 2020-10-07T11:41:24Z

packages/gatsby-source-contentful/src/normalize.js

+      // We need to add only root level resolvable (assets and entries)
+      // Derived nodes (markdown or JSON) will be recreated if needed.
+      const key = `${node.contentful_id}___${
+        node.sys.linkType || node.sys.type


@axe312ger this was the missing fix to make the tests pass

Hmm, I wonder how the nodes can be of type Link while entryList should contain a fully resolved Contentful API response:

https://github.com/gatsbyjs/gatsby/blob/contentful_drop_fixids/packages/gatsby-source-contentful/src/gatsby-node.js#L169-L186

pvdz · 2020-10-07T11:42:06Z

packages/gatsby-source-contentful/src/normalize.js

+      resolvable.add(
+        `${entry.sys.id}___${entry.sys.linkType || entry.sys.type}`
+      )
    })
  })
  assets.forEach(assetItem =>
-    resolvable.add(`${assetItem.sys.id}___${assetItem.sys.type}`)
+    resolvable.add(
+      `${assetItem.sys.id}___${assetItem.sys.linkType || assetItem.sys.type}`
+    )


I also applied it to these two, although it did not make a difference in the tests. @axe312ger is this a redundant change or good to keep?

See comment above. We should not have links in here at the first place.

It might be that the test fixtures are wrong/outdated.

pvdz · 2020-10-07T11:42:50Z

packages/gatsby-source-contentful/src/normalize.js

-        contentful_id: process.env.EXPERIMENTAL_CONTENTFUL_SKIP_NORMALIZE_IDS
-          ? entryItem.sys.id
-          : entryItem.sys.contentful_id,
+        contentful_id: entryItem.sys.id,


Tried dropping this entirely but that does break some tests so let's keep it for now. I think it makes sense for there to be a contentful_id to recognize atomic ids for content from Contentful

yes, we need to be able to identify content via contentful_id 👍

vladar · 2020-10-07T11:55:34Z

I am curious why did we have this fixId in the first place and why we don't need it anymore? Was it some workaround for old inference quirks?

pvdz · 2020-10-07T12:13:48Z

Yeah. I'm sorry I forgot the history already but the tldr is that it was useful at some point, no longer at some point, but nobody dared removing it. It's a huge drain on the sourcing time as it has to traverse the entire model, just like inference would.

wardpeet

In theory this is a breaking change if people for some reason have hard-coded ids.

pieh

It seems like there is few of things that are (or might be) breaking changes - this would need major version bump? (not a blocker, just let's try not to break user sites that might rely on those)

pieh · 2020-10-07T13:08:10Z

packages/gatsby-source-contentful/src/fetch.js

@@ -110,22 +109,6 @@ ${formatPluginOptionsForCLI(pluginConfig.getOriginalPluginOptions(), errors)}`)

  let contentTypeItems = contentTypes.items

-  if (process.env.EXPERIMENTAL_CONTENTFUL_SKIP_NORMALIZE_IDS) {
-    reporter.info(
-      `Skipping normalization of \`.id\`, this means \`sys\` objects will not get a \`.contentful_id\``


Are those sys objects accessible via schema? If so - that's breaking change

contentful_id will still available, see discussion

pvdz · 2020-10-07T13:22:35Z

Since not every node will get a contentful_id anymore, this is a potentially breaking change anyways. I'm not sure if there's anyone who depends on this, but considering that it's possible there probably is.
I think it was already the intention to release a major for this. It'll be up to Contentful to inform people that there's a major bump (not covered by ^). They are ultimately the code owner to this plugin.
In addition, there's another PR that will land soonish that also needs a major bump. But I suspect that one is a little further out.

axe312ger · 2020-10-08T11:50:54Z

Since not every node will get a contentful_id anymore, this is a potentially breaking change anyways. I'm not sure if there's anyone who depends on this, but considering that it's possible there probably is.
I think it was already the intention to release a major for this. It'll be up to Contentful to inform people that there's a major bump (not covered by ^). They are ultimately the code owner to this plugin.
In addition, there's another PR that will land soonish that also needs a major bump. But I suspect that one is a little further out.

This is correct. This is a potential breaking change has to be a major bump. The original idea was to release this one together with #25249, but the Rich Text fix will need more time to ready for release and will introduce some refactoring work for everyone using Rich Text.

Whats the usual strategy to inform the community about major bumps? I guess the Gatsby newsletter and blog?

I can make sure that Contentful will use their channels to inform people :)

pvdz · 2020-10-08T11:59:58Z

A major bump by itself is not a big deal, especially not here, because the most common way of depending on the package is by ^, which would not bump a major. So worst case, your customers are not getting the new updates until they realize they have to bump explicitly.

Ok, let's prepare a major bump. @axe312ger I assume there's no objections on your side since you haven't left any :)

axe312ger · 2020-10-08T12:02:29Z

packages/gatsby-source-contentful/src/normalize.js

+      // We need to add only root level resolvable (assets and entries)
+      // Derived nodes (markdown or JSON) will be recreated if needed.
+      const key = `${node.contentful_id}___${
+        node.sys.linkType || node.sys.type


Hmm, I wonder how the nodes can be of type Link while entryList should contain a fully resolved Contentful API response:

https://github.com/gatsbyjs/gatsby/blob/contentful_drop_fixids/packages/gatsby-source-contentful/src/gatsby-node.js#L169-L186

axe312ger · 2020-10-08T12:03:20Z

packages/gatsby-source-contentful/src/normalize.js

+      resolvable.add(
+        `${entry.sys.id}___${entry.sys.linkType || entry.sys.type}`
+      )
    })
  })
  assets.forEach(assetItem =>
-    resolvable.add(`${assetItem.sys.id}___${assetItem.sys.type}`)
+    resolvable.add(
+      `${assetItem.sys.id}___${assetItem.sys.linkType || assetItem.sys.type}`
+    )


See comment above. We should not have links in here at the first place.

It might be that the test fixtures are wrong/outdated.

* remove obsolete fixId function * there won't be contentful link objects on root level after the response got resolved

pvdz · 2020-10-15T12:10:32Z

@axe312ger was able to resolve outstanding issues. Provided tests hold this PR is good to go. It will require a major bump so let's make sure the current minor branch is stable before we do merge.

pieh · 2020-10-16T16:14:37Z

Merge commit message proposal:

perf(gatsby-plugin-contentful): drop the super expensive fixids cycle

BREAKING CHANGE: 
If you were relying on the `contentful_id` on the `sys` object in your queries  those are no longer exposed and you can safely change them to the `id` property on that `sys` object. 

If you were relying on the `id` property in the `sys` object, you should be aware that it is no longer "normalized". In particular, it will no longer get a 'c' prefixed to the id.

axe312ger · 2020-10-16T16:50:30Z

We published a new canary version. Feel free to give it a try:

npm i gatsby-source-contentful@next

It does contain the rich text fixes from #25249

pvdz · 2020-10-19T15:56:28Z

Published in gatsby-source-contentful@3.0.0 (together with gatsby@2.24.82, although that shouldn't matter so much)

perf(gatsby-plugin-contentful): drop the super expensive fixids cycle

88d496d

gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Oct 7, 2020

pvdz added topic: source-contentful Related to Gatsby's integration with Contentful topic: performance Related to runtime & build performance topic: scaling builds and removed status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer labels Oct 7, 2020

pvdz commented Oct 7, 2020

View reviewed changes

packages/gatsby-source-contentful/src/__tests__/__snapshots__/gatsby-node.js.snap Show resolved Hide resolved

pvdz commented Oct 7, 2020

View reviewed changes

update contentDigest values in snapshot

9f1fba9

wardpeet reviewed Oct 7, 2020

View reviewed changes

pieh reviewed Oct 7, 2020

View reviewed changes

wardpeet mentioned this pull request Oct 7, 2020

perf(gatsby-source-contentful): remove deprecated and slow fixids #25948

Closed

wardpeet added the breaking change If implemented, this proposed work would break functionality for older versions of Gatsby label Oct 8, 2020

axe312ger requested changes Oct 8, 2020

View reviewed changes

axe312ger mentioned this pull request Oct 15, 2020

contentful: improve dropping of fixids #27474

Merged

contentful: improve dropping of fixids (#27474)

1d33840

* remove obsolete fixId function * there won't be contentful link objects on root level after the response got resolved

Merge branch 'master' into contentful_drop_fixids

d7d7445

axe312ger approved these changes Oct 15, 2020

View reviewed changes

axe312ger mentioned this pull request Oct 16, 2020

Refactor Rich Text implementation in gatsby-source-contentful #25249

Merged

4 tasks

pvdz merged commit b316505 into master Oct 19, 2020

delete-merged-branch bot deleted the contentful_drop_fixids branch October 19, 2020 15:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf(gatsby-plugin-contentful): drop the super expensive fixids cycle #27318

perf(gatsby-plugin-contentful): drop the super expensive fixids cycle #27318

pvdz commented Oct 7, 2020

pvdz Oct 7, 2020

axe312ger Oct 8, 2020

pvdz Oct 7, 2020

axe312ger Oct 8, 2020

pvdz Oct 7, 2020

axe312ger Oct 8, 2020

vladar commented Oct 7, 2020

pvdz commented Oct 7, 2020

wardpeet left a comment

pieh left a comment

pieh Oct 7, 2020

axe312ger Oct 8, 2020

pvdz commented Oct 7, 2020

axe312ger commented Oct 8, 2020

pvdz commented Oct 8, 2020

axe312ger Oct 8, 2020

axe312ger Oct 8, 2020

pvdz commented Oct 15, 2020

pieh commented Oct 16, 2020

axe312ger commented Oct 16, 2020

pvdz commented Oct 19, 2020

perf(gatsby-plugin-contentful): drop the super expensive fixids cycle #27318

perf(gatsby-plugin-contentful): drop the super expensive fixids cycle #27318

Conversation

pvdz commented Oct 7, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladar commented Oct 7, 2020

pvdz commented Oct 7, 2020

wardpeet left a comment

Choose a reason for hiding this comment

pieh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pvdz commented Oct 7, 2020

axe312ger commented Oct 8, 2020

pvdz commented Oct 8, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pvdz commented Oct 15, 2020

pieh commented Oct 16, 2020

axe312ger commented Oct 16, 2020

pvdz commented Oct 19, 2020