feat(gatsby-source-shopify): download images option

[mrhut10]

Description

this PR aims to add a downloadImages option to the gatsby-source-plugin as described by @lukebennett88 in issue #23724
this prevents huge amounts of images being downloaded if you intended to use Shopify's CDN for images anyway greatly speeding up site build times.

This is my first PR to gatsby and neither @lukebennett88 or I feel it has had enough testing so hoping the community can test this implementation.

##Example Use
say you have a Shopify store with 500 products and 5 images per product. and you are happy to use Shopify's CDN (and transforming URLs) for the files. currently, the plugin will still download all of the images in this example that's 2500 images your downloading for no reason greatly slowing builds or timing out your build.

Documentation

new Options to gatsby-shopify-plugin

• downloadImages:
  • true (default) will download all shopify images
  • false: Shopify images won't download and in place local file will refer to a provided defaultImage ( so that graphQL types remain unchanged
• defaultImageUrl: default image url (if none provided will defaults to https://raw.githubusercontent.com/lukebennett88/gatsby-transformer-shopify-images/topic/shopify-option-downloadImages/plugins/gatsby-source-shopify/src/default.png

have changed the plugin readme file to reflect this @gatsbyjs/learning please read

Related Issues

should fix #23724

NOTE:
in the file "packages/gatsby-source/shopify/src/nodes.js" in the method call "downloadImageAndCreateFileNode"
do I need to do any cache invalidation in this file?
I have some concern that the referencing to cache with the fileNodeID may mean that when downloadImages is set back to true (from previously being false) it may try to use the cached file rather than the shopify images.

[mrhut10]

Any advice anyone what I can do to have this reviewed sooner?

[beamercola]

@mrhut10 Yes!! Been dreaming of this - same situation, many products (and the images are huge) so the whole app is failing right now and I can't boot it.

I'd love to test this - I'm having a hard time adding your branch into my project - do you have any pointers there?

edit: i ended up just plucking the shopify folder and pushing up to private repo as yarn dependency. this works perfectly - let's merge this bad boy i have like 4 projects i need this on!

[mrhut10]

@mrhut10 Yes!! Been dreaming of this - same situation, many products (and the images are huge) so the whole app is failing right now and I can't boot it.

I'd love to test this - I'm having a hard time adding your branch into my project - do you have any pointers there?

edit: i ended up just plucking the shopify folder and pushing up to private repo as yarn dependency. this works perfectly - let's merge this bad boy i have like 4 projects i need this on!

@beamercola reallly glad to hear you got it working and your keen for it.
thank @lukebennett88 for the idea which he talks about #23724

next project @lukebennett88 and I want to implement is a graphQL fragment which is compatible with gatsby-image that uses Shopify's CDN. would you also be keen on that in your projects @beamercola ??

[lukebennett88]

@beamercola Glad you got it working! It's nice to see other people wanting this functionality as well, I think it could be very useful for a lot of people.

[mrhut10]

hay @beamercola I've brought up to date before but not sure who's buttons we need to press to get it merged.

to help our case to get merged, can you (and anyone else) describe the pain your having / problem this will solve.

and on my end, I will relook at bring up to date next week and being it has been a while sence I've played with I think I better put it though its passes testing it on a demo project again.

[beamercola]

@mrhut10 My painpoint is I'm managing ~5 Shopify stores on Gatsby, some of which have multiple hundred products (each with ~5 images per product). Build time can be 20+ minutes. The annoyance is that I'm not using the local files/sharp for Gatsby, I'm relying on the CDN.

The modifications you made were perfect, it saved my butt with some angry clients after I sold them on the promise of Gatsby. But a quick look at the gatsby-source-shopify codebase it's looking very different from the initial fix and seems like this patch needs to be started all over.

I imagine there's a lot of Gatsby sites that include 5 products or so as a small "shop" section of the site. But the force downloading of all images is limiting to those who are excited about replacing Shopify with Gatsby for e-commerce.

Thanks for your patch, it's still going strong but getting nervous as the codebase is diverging a bit and I'm unable to update to newer versions without open heart surgery

[mrhut10]

@beamercola thank you so much for sharing your pain, sounds like your describing the exact circumstance that this branch / feature was made for when @lukebennett88 first thought it up.

:) yay.

but I haven't look much codebase with the changes they've recently done. I'll certainly try to have a look at it next week.
happy to walk you though what I find, and maybe an opportunity to pair program out a solution if you or anyone wanted to as well.

[KyleAMathews]

Hi all. So sorry this hasn't gotten in yet. It's a good change.

I tried merging master locally and it merged cleanly so it looks like this can get it easily still. @mrhut10 if you could give me write access to the PR, I can push the changes + do some testing and help get this in.

[mrhut10]

Hi all. So sorry this hasn't gotten in yet. It's a good change.

I tried merging master locally and it merged cleanly so it looks like this can get it easily still. @mrhut10 if you could give me write access to the PR, I can push the changes + do some testing and help get this in.

Gladly and sounds exciting, @KyleAMathews
I'll get you the wire access to the repo tonight ( Australia time)

[mrhut10]

there you go @KyleAMathews I've added you as a collaborator on my fork of the project :)
thank you for looking at it.

[KyleAMathews]

Merged in master. We'll see how tests go.

[mrhut10]

thank you @KyleAMathews I'll let you know once I make those changes :)

[mrhut10]

just having a play with this now, also noticed in the future we might want to support turning off images for particular types of documents, i.e. Product Images, Collection Images, and Article Images if theres a needs for it could selectively turn this off. But I guess see if that gets requested after this base feature is in master.

[mrhut10]

Hay @KyleAMathews I've committed those changes from your review.

• Removed the pluginOptions singleton class
• Removal of defaultUrl or images
• if !downloadImages then early return (setting the localFile to undefined) before downloading the image

[mrhut10]

testing I've already done @KyleAMathews

Setup Environment

• clone gatsby store
• resolve the gatsby-source-shopify plugin locally to my altered and build branch

Test Cases

1. not setting config.downloadImages ( then will default to being set to true, i.e. will download images)
2. setting config.downloadImages to false - removes the localFile from graphQl types on product images.
   
3. setting config.downloadImages to true - adds the localFile from graphQl types on product images
   

[mrhut10]

@beamercola / @GooBall / @lukebennett88 are any of you already using shopify CDN for images?
and have you managed to still use the gatsby-image component or just using image components directly?

once this is merged thinking it would be good to look at the possibility to:

1. create a few blog posts showing people how to use the shopify cdn with this option to greatly decrease build times of sites.
2. potentially look at creating a gatsby-transformer-shopifyCdnImages which makes a gatsby-image compatible data, so people can still define the image sizes in the graphQL query, and get back a gatsby-image compatible data so that they can use the Component thats familiar and often recommended for Gatsby projects.

[lukebennett88]

Hey @mrhut10 I haven't looked into this for a while.

To resize an image I would append _${width}x just before the file extension.

e.g.:

Original:
https://cdn.shopify.com/s/files/1/0513/3785/5157/products/2711.jpg

Resized:
https://cdn.shopify.com/s/files/1/0513/3785/5157/products/2711_200x.jpg

I wrote a crude function to resize them which was good enough for how I was using it:

function resizeShopifyImage({ url, width }) {
  let fileExtension = '.jpg';
  let index = url.indexOf('.jpg');
  if (index === -1) {
    index = url.indexOf('.png');
    fileExtension = '.png';
  }
  if (index !== -1) {
    const firstChunk = url.slice(0, index);
    return firstChunk.concat(`_${width}x${fileExtension}`);
  }
  return url;
}

export { resizeShopifyImage };

I know at the time I there was an issue with the CDN — if I requested an image that was larger that the original image, the image wouldn't load. This doesn't seem to be the case anymore so a using this with Gatsby Image should probably work (I'm fairly certain that the base64 image isn't required since there is a Gatsby Image fragment with no base64 encoded image:
https://www.gatsbyjs.com/plugins/gatsby-image/?=gatsby-image#gatsby-transformer-sharp

One other thing to consider is that the Gatsby Image team are working on a replacement for Gatsby Image — Gatsby Plugin Image so it's probably a good idea to make it compatible with that so we don't have to redo it later.

[KyleAMathews]

One other thing to consider is that the Gatsby Image team are working on a replacement for Gatsby Image — Gatsby Plugin Image so it's probably a good idea to make it compatible with that so we don't have to redo it later.

👍 to this. Here's an example of how source plugins can do the integration — #28236

[mrhut10]

@KyleAMathews is there anyone we can request from gatsbyjs/documentation to review?

[KyleAMathews]

Thanks! Sorry again for the slow merge.

[KyleAMathews]

I've got the power ;-)

[mrhut10]

Yay!!!! so excited!! thank you @KyleAMathews

@beamercola and/or @lukebennett88 can I use one of your sites that has too many shopify images in a blog post showing the build time performance increase? or/and interested in writing one yourself? (I still need to set up a blog or medium or something)

[beamercola]

@KyleAMathews for prez

@mrhut10 yeah, happy to!

[lukebennett88]

Yay!!!! so excited!! thank you @KyleAMathews

@beamercola and/or @lukebennett88 can I use one of your sites that has too many shopify images in a blog post showing the build time performance increase? or/and interested in writing one yourself? (I still need to set up a blog or medium or something)

So happy to see this merged!
I'll update the source plugin on GLF Online tomorrow and most the results.

[lukebennett88]

How do the npm releases work?
Are they updated on a schedule? Hoping to test this tomorrow if it's released by then 🤞

[mrhut10]

@lukebennett88 its possible its automated, but otherwise I believe they would have to increment the version number in the package.json and then do a deployment to npm.

in the meanwhile to run locally in your dev environment

1. could copy the files from gatsby repo packages/gatsby-source-shopify folder into your project under the folder /packages/gatsby-source-shopify
2. then cd into that folder and run npm install && npm build
3. then update your gatsby-config.json file to reflect that you want to resolve the plugin locally

// gatsby-config.js
{
      resolve: require.resolve(`./packages/gatsby-source-shopify`),
      options: {
        shopName: 'gatsby-swag',
        accessToken: process.env.SHOPIFY_ACCESS_TOKEN,
        downloadImages: true
      }
 },

If you want to run in production, you will need to do

1. git add and commit the files in packages/gatsby-source-shopify (note the gitignore file they provide should already ignore the files that the build command generates.
2. update your build command to also build the package i.e. in your package.json scripts "build": cd packages/gatsby-source-shopify && npm install && npm build && cd ../.. && gatsby build" (don't quote me on that one but will give you the idea hopefully)

please let me know if you want a hand to try / set up

[lukebennett88]

Thanks @mrhut10, yeah I know I could get it working manually, but didn't want to go through the extra work if the package is likely to be updated soon.

Appreciate the write-up though, and thanks for going the last mile to finally get this thing merged in. So happy 🎉

[mrhut10]

@lukebennett88 looks like its been tagged and deployed under the tag 3.9.0-next.1 so you should be able to npm install that tag and use this feature with that version tag.

//npm
npm install gatsby-source-shopify@3.9.0-next.1
// yarn
yarn add gatsby-source-shopify@3.9.0-next.1

likewise you could always ask @beamercola to redeploy his npm version which he was using for his sites.
npm link