fix(gatsby-plugin-utils): path pieces too long and url safe base64 encoding

[veryspry]

Description

The PR solves two bugs:

1. When downloading images with the gatsbyImage resolver locally, sometimes base64 encoded pieces would exceed system limits. The fix is to move to query parameters and use md5 hashes for the base64 pieces when an image is downloaded and saved to disk during build time.
2. The base64 spec includes forward slashes making it unsuitable for use in URL's, with the query param approach we can ditch base64

Related Issues

Fixes #35126

[wardpeet]

Sharing a bit on why I changed the code here. Image Url & args can indeed be any value like @veryspry did. Instead of creating our own md5 hash, we use createContentDigest (which is also an md5 hash but native to Gatsby). For url and args we will use query string parameters as they can be any length and keep code between image-cdn and local images the same.

[TylerBarnes]

@wardpeet for files on disk (with image CDN turned off) wont the query args not work?

EDIT: I see you have an md5 in the file path with the args - that makes sense 👍 I can see how it works now

[LekoArts]

Judging by the tests this looks good

[LiamMartens]

Just bumping this because we are running into this issue as well 👍

[davidpaulsson]

If I want to test this without the ERROR ENAMETOOLONG: name too long, mkdir error happening, is there a specific pkg or version I should set a resolution to?

[LekoArts]

@davidpaulsson You can use the @next versions

[thekage91]

Hi, I run into this issue during the building of my site. I retrieve datas from WordPress and the process fails during gatsby.IMAGE_CDN job which return this error

UNHANDLED REJECTION ENAMETOOLONG: name too long, mkdir '/Users/davide/Workspace/GitHub-Repo/ecommerce-scraper-frontend/packages/blog/public/_gatsby/image/aHR0cHM6Ly9yZXMuY2xvdWRpbmFyeS5jb20vcHJpY3lvL2ltYWdlcy9mX2F1dG8scV9hdXRvL3YxNjQ0NzAxMzExL3ZpbmlwZXJ
iYWNjby1tYXRyaWNlLWNvbmZyb250by1wb3NpemlvbmFtZW50aS1nZW5uYWlvLXZzLWx1Zy92aW5pcGVyYmFjY28tbWF0cmljZS1jb25mcm9udG8tcG9zaXppb25hbWVudGktZ2VubmFpby12cy1sdWcucG5nP19pPUFB'



  WorkerError: ENAMETOOLONG: name too long, mkdir '/Users/davide/Workspace/GitHub-Repo/ecommerce-scraper-frontend/packages/blog/public/_gatsby/image/aHR0cHM6Ly9yZXMuY2xvdWRpbmFyeS5jb20vcHJpY3lvL2ltYWdlcy9mX2F1dG8scV9hdXRvL3YxNjQ0NzAxMzExL3ZpbmlwZXJiYWNj
  by1tYXRyaWNlLWNvbmZyb250by1wb3NpemlvbmFtZW50aS1nZW5uYWlvLXZzLWx1Zy92aW5pcGVyYmFjY28tbWF0cmljZS1jb25mcm9udG8tcG9zaXppb25hbWVudGktZ2VubmFpby12cy1sdWcucG5nP19pPUFB'
  
  - manager.ts:311 enqueueJob
    [ecommerce-scraper-frontend]/[gatsby]/src/utils/jobs/manager.ts:311:21

I have "gatsby": "^4.14.0-next.0" version but the problem still persist :(

Can anyone help me?
Thank you

[veryspry]

Hi, I run into this issue during the building of my site. I retrieve datas from WordPress and the process fails during gatsby.IMAGE_CDN job which return this error

UNHANDLED REJECTION ENAMETOOLONG: name too long, mkdir '/Users/davide/Workspace/GitHub-Repo/ecommerce-scraper-frontend/packages/blog/public/_gatsby/image/aHR0cHM6Ly9yZXMuY2xvdWRpbmFyeS5jb20vcHJpY3lvL2ltYWdlcy9mX2F1dG8scV9hdXRvL3YxNjQ0NzAxMzExL3ZpbmlwZXJ
iYWNjby1tYXRyaWNlLWNvbmZyb250by1wb3NpemlvbmFtZW50aS1nZW5uYWlvLXZzLWx1Zy92aW5pcGVyYmFjY28tbWF0cmljZS1jb25mcm9udG8tcG9zaXppb25hbWVudGktZ2VubmFpby12cy1sdWcucG5nP19pPUFB'



  WorkerError: ENAMETOOLONG: name too long, mkdir '/Users/davide/Workspace/GitHub-Repo/ecommerce-scraper-frontend/packages/blog/public/_gatsby/image/aHR0cHM6Ly9yZXMuY2xvdWRpbmFyeS5jb20vcHJpY3lvL2ltYWdlcy9mX2F1dG8scV9hdXRvL3YxNjQ0NzAxMzExL3ZpbmlwZXJiYWNj
  by1tYXRyaWNlLWNvbmZyb250by1wb3NpemlvbmFtZW50aS1nZW5uYWlvLXZzLWx1Zy92aW5pcGVyYmFjY28tbWF0cmljZS1jb25mcm9udG8tcG9zaXppb25hbWVudGktZ2VubmFpby12cy1sdWcucG5nP19pPUFB'
  
  - manager.ts:311 enqueueJob
    [ecommerce-scraper-frontend]/[gatsby]/src/utils/jobs/manager.ts:311:21

I have "gatsby": "^4.14.0-next.0" version but the problem still persist :(

Can anyone help me? Thank you

Hey @thekage91 👋

For some extra context, this PR made a change to md5 hash some of the image path segments via the gatsby createContentHash helper function instead of base64 encode them.

Taking a look at the path in your error:

'/Users/davide/Workspace/GitHub-Repo/ecommerce-scraper-frontend/packages/blog/public/_gatsby/image/aHR0cHM6Ly9yZXMuY2xvdWRpbmFyeS5jb20vcHJpY3lvL2ltYWdlcy9mX2F1dG8scV9hdXRvL3YxNjQ0NzAxMzExL3ZpbmlwZXJ
iYWNjby1tYXRyaWNlLWNvbmZyb250by1wb3NpemlvbmFtZW50aS1nZW5uYWlvLXZzLWx1Zy92aW5pcGVyYmFjY28tbWF0cmljZS1jb25mcm9udG8tcG9zaXppb25hbWVudGktZ2VubmFpby12cy1sdWcucG5nP19pPUFB'

I extract the last piece:

aHR0cHM6Ly9yZXMuY2xvdWRpbmFyeS5jb20vcHJpY3lvL2ltYWdlcy9mX2F1dG8scV9hdXRvL3YxNjQ0NzAxMzExL3ZpbmlwZXJiYWNjby1tYXRyaWNlLWNvbmZyb250by1wb3NpemlvbmFtZW50aS1nZW5uYWlvLXZzLWx1Zy92aW5pcGVyYmFjY28tbWF0cmljZS1jb25mcm9udG8tcG9zaXppb25hbWVudGktZ2VubmFpby12cy1sdWcucG5nP19pPUFB

This will either be a base64 encoded version of the source image URL OR an md5 hash of the image source URL depending on the Gatsby version you are on.

If I base64 decode the string:

echo "aHR0cHM6Ly9yZXMuY2xvdWRpbmFyeS5jb20vcHJpY3lvL2ltYWdlcy9mX2F1dG8scV9hdXRvL3YxNjQ0NzAxMzExL3ZpbmlwZXJiYWNjby1tYXRyaWNlLWNvbmZyb250by1wb3NpemlvbmFtZW50aS1nZW5uYWlvLXZzLWx1Zy92aW5pcGVyYmFjY28tbWF0cmljZS1jb25mcm9udG8tcG9zaXppb25hbWVudGktZ2VubmFpby12cy1sdWcucG5nP19pPUFB" | base64 --decode

The operation is successful and I get this back which tells me that the version of gatsby you have installed does not include this fix.

https://res.cloudinary.com/pricyo/images/f_auto,q_auto/v1644701311/viniperbacco-matrice-confronto-posizionamenti-gennaio-vs-lug/viniperbacco-matrice-confronto-posizionamenti-gennaio-vs-lug.png?_i=AA%                                        

Can you try using the explicit @next version of Gatsby? I'm wondering if the explicit version you used, ^4.14.0-next.0, just doesn't have this fix included.

Let me know if this helps and if not, feel free to ping me back here and hopefully we can get this sorted out 🙂