Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve arm64 build times through cross-compilation #709

Closed
4 tasks
svrnm opened this issue Jan 22, 2023 · 11 comments
Closed
4 tasks

Improve arm64 build times through cross-compilation #709

svrnm opened this issue Jan 22, 2023 · 11 comments
Labels
enhancement New feature or request

Comments

@svrnm
Copy link
Member

svrnm commented Jan 22, 2023

Feature Request

Is your feature request related to a problem?

As stated in this comment by @austinlparker the build for arm64 takes ~4hrs right now. This time hopefully should improve due to the plenty improvements that now exist for the Dockerfiles, but will probably still be significant (e.g. the rust image took 1+hrs and is not at all optimized)

Describe the solution you'd like:

I suggest making use of cross compilation. I did a tiny experiment with the checkoutservice and it improved the build times from ~7mins to ~2mins:

https://github.com/svrnm/playground/actions/runs/3981194659
https://github.com/svrnm/playground/actions/runs/3981102723

The dockerfiles need to be changed for that as outlined in this blog post, e.g. here's my checkout service Dockerfile for cross compilation:

https://github.com/svrnm/playground/blob/main/src/checkoutservice/Dockerfile

The good thing is, that those changes don't need to be introduced at once and also not for all services. My suggestion is that we pick the ones which take the longest:

  • Currency
  • Payment
  • Quote
  • Shipping

What do you all think?

@svrnm svrnm added the enhancement New feature or request label Jan 22, 2023
@cartersocha
Copy link
Contributor

FYI @open-telemetry/demo-approvers

super interesting and worth pursuing I think

@austinlparker
Copy link
Member

I'm not entirely sure this is going to help with the worst-case stuff. It's a nice reduction in time, but since builds happen monolithically (every image is built on release, we don't do builds for specific changed containers during PRs, etc) saving time on go doesn't really help unless we get the really slow stuff broken up and we can't cross-compile the dependency chains for grpc afaik?

@svrnm
Copy link
Member Author

svrnm commented Jan 23, 2023

@austinlparker this also works for C++/Rust/..., I just picked Go as an example because it was kind of easy. I played with the rust stuff a little bit, but have no results yet. In theory you should be able to cross-compile everything (including the grpc dependency chain), so the ideal setup is to have a build stage that does 99% of the heavy lifting and a release stage that just copies files out of that

@austinlparker
Copy link
Member

Hm, when I tried doing the cross-compilation stuff I wasn't able to get it to work on Rust (and the C++ side was really frustrating), maybe you'll have better luck.

@svrnm
Copy link
Member Author

svrnm commented Jan 23, 2023

Hm, when I tried doing the cross-compilation stuff I wasn't able to get it to work on Rust (and the C++ side was really frustrating), maybe you'll have better luck.

I will at least give it a try, if I succeed awesome, if not this issue here will be a reminder of my hubris and a warning for future contributors.

@svrnm
Copy link
Member Author

svrnm commented Jan 25, 2023

Quick update on that, I did a trial with the current state of all Dockerfiles & one more with the patched Dockerfile for the shipping-service. Without testing functionality I can report that building for AMD64 & ARM64 takes

  • 44 minutes with what we have as of today
  • 20 minutes with the cross-compilation for shipping service. With that the featureflag service is taking the longest, currencyservice comes second.

To be honest, I am not sure if adding #715 is worth the hassle, 44min sounds OK-ish to me.

@austinlparker
Copy link
Member

Honestly 20 minutes is a lot closer to where we wanted to be... I believe our target is 15m. I agree that the new Dockerfile is pretty ugly, but it shouldn't need to be putzed with a lot, right?

@cartersocha
Copy link
Contributor

A decreased build time is more useful to project contributors imo.

I barely look at the docker files personally but have to build things a lot and the Apple silicon issue isn't going away

+1 on build time over complexity

@svrnm
Copy link
Member Author

svrnm commented Jan 25, 2023

Ok, I was not aware of that 15min target ;-)

I agree that the new Dockerfile is pretty ugly, but it shouldn't need to be putzed with a lot, right?

True, and it still can be converted to something more simple if we run into any issues.

I did some reading yesterday on branching in Dockerfiles, so maybe there's an option to tidy up things, will give it another look later this week

@puckpuck
Copy link
Contributor

puckpuck commented Mar 9, 2023

Building locally for a single architecture is much faster now.
Intel MacBook Pro (2019): 12m45s
M2 Max MacBook (2023): 4m41s

The 1.3.1 release, which is multiple architectures for amd64 and arm64, took 24m23s.

All these build times are reasonable. Should we close this issue?

@svrnm
Copy link
Member Author

svrnm commented Mar 9, 2023

looks good enough for me to have this closed

@puckpuck puckpuck closed this as completed Mar 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants