feat(gatsby-core-utils): Add retry on HTTP status codes to fetchRemoteFile

[axe312ger]

We currently do not retry on certain status codes and network errors. This can cause issues when building projects that use a lot of remote assets that need to be fetched.

This replaces #30391.

Status:

• add retry for error status codes
• add retry for network errors
• integrate with all plugins that use fetchRemoteFile

[axe312ger]

@wardpeet as you suggested, I moved the logic from my wrapper into the error handler of the stream.

I struggle to add an actual timeout to the retry.

Also I am still not sure how and if we should simulate generic network errors, but for sure we should try to retry them.

[axe312ger]

Out of scope for this.. but... what if we send a user-agent that contains GatsbyJS and the current version? Could imagine that would increase the visibility in logs, and might even help in support tickets.

[wardpeet]

Nice!

[njbmartin]

This is sadly blocking us from being able to upgrade to Gatsby v4 as our build fails trying to download images from Contentful. Happens randomly but consistently after downloading 250+ images for us in the dev environment.

Edit: A fun read about unexplained connection resets in Docker - https://tech.xing.com/a-reason-for-unexplained-connection-timeouts-on-kubernetes-docker-abd041cf7e02

[axe312ger]

@njbmartin thank you for the link. This is currently our highest priority, I want to fix this in 2021!

[axe312ger]

The old version of gatsby-source-contentful was reducing this issues by limiting the parallel downloads to 100 at the same time. It was not perfect, but drastically reduced the number of required retries.

[wardpeet]

Awesome! This fixes so many things! 🎉

[axe312ger]

Ohh yeah! :)

[LekoArts]

Successfully published:
 - babel-plugin-remove-graphql-queries@4.1.1
 - babel-preset-gatsby@2.1.1
 - gatsby-cli@4.1.2
 - gatsby-core-utils@3.1.1
 - gatsby-page-utils@2.1.1
 - gatsby-plugin-benchmark-reporting@2.1.1
 - gatsby-plugin-gatsby-cloud@4.1.1
 - gatsby-plugin-image@2.1.1
 - gatsby-plugin-manifest@4.1.2
 - gatsby-plugin-mdx@3.1.2
 - gatsby-plugin-offline@5.1.2
 - gatsby-plugin-page-creator@4.1.2
 - gatsby-plugin-preload-fonts@3.1.1
 - gatsby-plugin-sharp@4.1.2
 - gatsby-plugin-typescript@4.1.1
 - gatsby-recipes@1.1.1
 - gatsby-remark-images@6.1.2
 - gatsby-source-contentful@6.1.2
 - gatsby-source-drupal@5.1.2
 - gatsby-source-filesystem@4.1.1
 - gatsby-source-graphql@4.1.1
 - gatsby-source-shopify@6.1.2
 - gatsby-source-wordpress@6.1.1
 - gatsby-telemetry@3.1.1
 - gatsby-transformer-remark@5.1.2
 - gatsby-transformer-sqip@4.1.2
 - gatsby@4.1.3