feat(gatsby-source-drupal): Use the collection count from JSON:API extras to enable parallel API requests for cold builds #32883
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tras to construct URLs Otherwise, we have to wait to start querying each page until the previous one finishes. This change lets us query all pages in parallel. So instead of fetching one collection page at a time, we can fetch up to the maximum concurrency (default 20). For a test site with ~3200 entities, this PR dropped sourcing time from ~14s to 4s.
gatsbot
bot
added
the
status: triage needed
KyleAMathews
changed the title
drupalninja
previously approved these changes
Aug 23, 2021
KyleAMathews
added
topic: source-drupal
|
@KyleAMathews - should we include an addition to the readme to help people discover this update? ie - push people to enable the proper JSON API Extras settings? |
wardpeet
reviewed
Aug 23, 2021
|
Yeah definitely. I'll add it once we do some more real-world testing to verify this change. |
KyleAMathews
changed the title
smthomas
previously approved these changes
Aug 26, 2021
wardpeet
pushed a commit
that referenced
this pull request
Aug 27, 2021
…tras to enable parallel API requests for cold builds (#32883) * feat(gatsby-source-drupal): Use the collection count from JSON:API extras to construct URLs Otherwise, we have to wait to start querying each page until the previous one finishes. This change lets us query all pages in parallel. So instead of fetching one collection page at a time, we can fetch up to the maximum concurrency (default 20). For a test site with ~3200 entities, this PR dropped sourcing time from ~14s to 4s. * use new browser-based URL parser * Comment code more * Use the page size the site has set instead of assuming 50 * Use the original type that's set as that's always there * Log out updates while sourcing * Encourage people to enable this setting in the README * Update gatsby-node.js
This was referenced Aug 31, 2021
This was referenced Oct 19, 2024
This was referenced Oct 20, 2024
This was referenced Oct 21, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Otherwise, we have to wait to start querying each page until the previous one finishes. This change lets us query all collection pages in parallel. So instead of fetching one collection page at a time, we can fetch up to the maximum concurrency (default 20).
For a test site with ~3200 entities and a warm Drupal cache (and no CDN cache), this PR dropped sourcing time from ~14s to 4s.
On a very large production Drupal site (~600k entities). Fetching time for a cold build dropped from 2 hours to 30 minutes 🚀
TODOs