Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci(stage-build): rsync with -j html,json,txt #8549

Merged
merged 1 commit into from
Mar 31, 2023
Merged

ci(stage-build): rsync with -j html,json,txt #8549

merged 1 commit into from
Mar 31, 2023

Conversation

caugner
Copy link
Contributor

@caugner caugner commented Mar 31, 2023

Summary

Part of MP-315.

Problem

The GCP deployment is still slow, and copying client/build takes longer than it used to take in AWS.

Solution

Accelerate the storage copy using the -j option:

Applies gzip transport encoding to any file upload whose extension matches the -j extension list. This is useful when uploading files with compressible content (such as .js, .css, or .html files) because it saves network bandwidth while also leaving the data uncompressed in Cloud Storage. (Source)

Note: json, html and txt are by far the most frequent text formats in the build.

% find . -type f | sed -e 's/.*\.//' | sort | uniq -c | sort -nr
81115 json
52112 html
40502 txt
2146 png
 276 jpg
 218 svg
  85 gif
  32 map
  21 js
  12 css
   9 gz
   2 xml
   2 woff2
   1 jpeg
   1 ico

How did you test this change?

Uses compression for text files.
@caugner caugner marked this pull request as draft March 31, 2023 18:07
@caugner caugner temporarily deployed to stage March 31, 2023 18:08 — with GitHub Actions Inactive
@caugner
Copy link
Contributor Author

caugner commented Mar 31, 2023

For my tests above I used a 4-core large runner, but I want to ensure that the stage-build time does not regresses with the standard 2-core runners either, so I have marked this PR as draft while this stage-build is running.

@caugner caugner marked this pull request as ready for review March 31, 2023 19:17
@caugner
Copy link
Contributor Author

caugner commented Mar 31, 2023

@bkochendorfer Wow, 42m for build + sync to AWS + sync to GCP!

@caugner caugner merged commit d784c7f into main Mar 31, 2023
@caugner caugner deleted the gsutil-rsync-j branch March 31, 2023 19:33
wbamberg added a commit to wbamberg/yari that referenced this pull request Apr 4, 2023
* upstream/main: (251 commits)
  chore(deps-dev): bump typescript from 4.9.5 to 5.0.3 in /client/pwa (mdn#8547)
  chore(deps-dev): bump typescript from 4.9.5 to 5.0.3 (mdn#8543)
  chore(deps-dev): bump postcss-preset-env from 8.2.0 to 8.3.0 (mdn#8561)
  chore(deps-dev): bump @types/react from 18.0.31 to 18.0.32 (mdn#8560)
  chore(advertising): update copies (mdn#8562)
  fix(client): place hash before headings (mdn#8532)
  chore(deps-dev): bump ts-jest from 29.0.5 to 29.1.0 (mdn#8559)
  chore(deps-dev): bump black from 23.1.0 to 23.3.0 in /deployer (mdn#8555)
  chore(deps-dev): bump webpack-dev-server from 4.13.1 to 4.13.2 (mdn#8558)
  chore(deps-dev): bump swr from 2.1.1 to 2.1.2 (mdn#8556)
  chore(deps): bump boto3 from 1.26.99 to 1.26.104 in /deployer (mdn#8557)
  chore(deps-dev): bump black in /testing/integration (mdn#8554)
  ci(stage-build): rsync with -j html,json,txt (mdn#8549)
  ci(stage-build): rsync with -c (mdn#8548)
  chore(deps-dev): bump react-router and react-router-dom from 6.9.0 to 6.10.0 (mdn#8534)
  chore(deps-dev): bump @babel/preset-env from 7.20.2 to 7.21.4 (mdn#8544)
  chore(deps-dev): bump tailwindcss from 3.3.0 to 3.3.1 (mdn#8546)
  chore(deps): bump @mdn/browser-compat-data from 5.2.46 to 5.2.47 (mdn#8545)
  chore(deps-dev): bump @babel/core from 7.21.3 to 7.21.4 (mdn#8542)
  fix(docs): make icons in good/bad examples float (mdn#8531)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant