Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(gatsby-source-contentful): maintain back reference map between runs #37442

Merged
merged 2 commits into from
Jan 12, 2023

Conversation

pieh
Copy link
Contributor

@pieh pieh commented Jan 11, 2023

Description

Some Contentful delta sync updates are currently stripping previously created back reference links when they shouldn't.

This happen because when we create a link map (which we call foreighReferenceMap in the plugin) we only used entries that were created or updated since last sync. If sync didn't update referencing entry, then updated entries would lose backreferences to referecing entry.

To fix the problem this PR maintains state of foreighReferenceMap between sourceNodes runs keeping it fully saturated at all times. Traversing everything in each sourceNodes runs would be expensive so foreignReferenceMap now also have sort of "delta updates":

  • I introduced foreignReferenceMapState which is object with
    • backLinks (which is exact same structure as foreignReferenceMap, so can be used as-is in later stages of sourcing, without having to refactor bunch of code in the plugin)
    • links which is somewhat reverse mapping - keys are contentful's id and values is array of field names in backLinks - this is done to be able to do guided deletions without nested for loops if we would only have backLinks ( similar to what perf(gatsby): fix performance regression with query dependency cleaning #28032 did for increased perf to similar data structure problem)
  • on update
    • I clear out references for any of created/updated OR deleted entries (new)
    • I traverse created/updated entries to regenerate links data for it

Related Issues

[ch-59240]

@gatsbot gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Jan 11, 2023
@pieh pieh added topic: source-contentful Related to Gatsby's integration with Contentful and removed status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer labels Jan 11, 2023
@pieh pieh force-pushed the contentful-untouched-node-backref branch from c9315d2 to fcd6f11 Compare January 11, 2023 18:41
@axe312ger axe312ger self-requested a review January 12, 2023 10:40
@pieh pieh marked this pull request as ready for review January 12, 2023 11:31
Copy link
Collaborator

@axe312ger axe312ger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't see any logic issues, plus tests are green.

This should definitely improve the reliability of the plugin!

Thank you!

@pieh pieh merged commit 76f979c into master Jan 12, 2023
@pieh pieh deleted the contentful-untouched-node-backref branch January 12, 2023 14:31
@axe312ger
Copy link
Collaborator

🥂

pieh added a commit that referenced this pull request Jan 12, 2023
…ns (#37442)

* test(gatsby-source-contentful): add test case

* fix(gatsby-source-contentful): maintain back reference map between runs

(cherry picked from commit 76f979c)
pieh added a commit that referenced this pull request Jan 12, 2023
…ns (#37442)

* test(gatsby-source-contentful): add test case

* fix(gatsby-source-contentful): maintain back reference map between runs

(cherry picked from commit 76f979c)
pieh added a commit that referenced this pull request Jan 12, 2023
…ns (#37442) (#37455)

* test(gatsby-source-contentful): add test case

* fix(gatsby-source-contentful): maintain back reference map between runs

(cherry picked from commit 76f979c)

Co-authored-by: Michal Piechowiak <misiek.piechowiak@gmail.com>
pieh added a commit that referenced this pull request Jan 12, 2023
…ns (#37442) (#37456)

* test(gatsby-source-contentful): add test case

* fix(gatsby-source-contentful): maintain back reference map between runs

(cherry picked from commit 76f979c)

Co-authored-by: Michal Piechowiak <misiek.piechowiak@gmail.com>
@pieh
Copy link
Contributor Author

pieh commented Jan 13, 2023

Published in:

  • gatsby-source-contentful@7.23.1
  • gatsby-source-contentful@8.4.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: source-contentful Related to Gatsby's integration with Contentful
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants