Caching issues with --prefix-paths

[marcus13371337]

Hello,

In our current setup, some of our pages are built with gatsby and hosted on s3. For these pages we have added redirects for some specific urls in our web proxy. Since we are doing that we can't put the assets at root (as example /styles.785b0331f15c65aecdef.css needs to be /_gatsby/styles.785b0331f15c65aecdef.css), which is why we are using the --path-prefix with _gatsby, which we have added a rule for in our external proxy.

However, we still want to publish/host our assets in the same bucket as we use for the html-pages. I noticed that you ignore all assets files if the --prefix-paths is set. We kind of found a hack to get around this, which is to take all the asset files and put them in a directory inside the build directory (named _gatsby) and your plugin will manage the upload beautifully.

However, it seems as we have some cache problems with the page-data.json-files. It seems as the metadata for the cache control is not correctly set. I guess it's because of this rule, which won't match since the page-data is prefixed with /_gatsby/page-data/**/**.json

    'page-data/**/**.json': {
        CacheControl: 'public, max-age=0, must-revalidate',
    },

Do you have any brilliant ideas on how we can solve the issue?

[YoshiWalsh]

Hi, thanks for reaching out.

Please note that prefixes are something that we're still working on. They are a pain point for a lot of our users. (See gatsbyjs/gatsby#27988, #208, and #24).

Based on your description, I think you want to upload objects to S3 with the _gatsby prefix, but viewers of your website would still navigate to the root of the domain, (_gatsby should not appear in the address bar), is that correct? If so, instead of using Gatsby's --path-prefix I recommend that you use our plugin's bucketPrefix option.

If I misunderstood and you want the _gatsby to appear in the URL, I think you'll have to wait until we get a response from the Gatsby Core team.

[marcus13371337]

No, we are incrementally migrating our site to gatsby. So the bucket has the same structure as of being served from the root. So for example if someone requests /our-gatsby-pages/* we will point them to our s3-bucket (and the .html-page lives in the directory /our-gatsby-pages/xxx in the bucket), but if someone requests /our-other-pages we will point them to our legacy solution.

Since we can't add such rederiction rules for each file that gatsby generates, we used the --prefix-paths with assetsPrefix: _gatsby which makes each static .html-page to request all the .js/.css files from: /_gatsby/something.123.css instead of just /something.123.css. So yes and no, we are fine with /_gatsby visible in the url for assets, but not for our pages

[YoshiWalsh]

Ah, so you're using assetPrefix and not pathPrefix? (Both use the --prefix-paths option)

We don't current have native support for this. But we do provide the ability for you to define your own rules for parameters applied to objects, see here. You should be able to use this to ensure the correct caching rule is applied.

{
    "_gatsby/page-data/**/**.json": {
        "CacheControl": "public, max-age=0, must-revalidate",
    }
}

Keep in mind that metadata-only changes are not published for existing objects, this is a known limitation. You will need to delete any existing objects from the bucket prior to deployment in order for the new metadata to be applied.

[marcus13371337]

Ah cool! Didn't know about pathPrefix, but that's right we are only using the assetPrefix.

Thanks, that looks promising will test that out.

I guess this kind of ends with a feature request then, which is that it should be super cool to support a local assetPrefix, which gets published in the same bucket natively!

[YoshiWalsh]

assetPrefix is kind of an interesting one, because it's really intended for hosting your assets on a separate subdomain/domain entirely. In that use case assets would most likely be uploaded to a different bucket.

Maybe an option would be to allow configuring the upload of pages and assets seperately. You could choose to use this plugin to deploy assets, pages, or both. If both, you could upload them to the same bucket, or to two different buckets.

Anyway, I think it's pretty niche so I don't think we'll prioritise it, but if you want to submit a PR for this functionality I'd be happy to review it.

[marcus13371337]

Cool! Totally agree with you, haven't seen that the gatsby docs suggesting to use the assetPrefix for the same domain so maybe that's a hack by us, but it's working 😀