Improve arm64 build times through cross-compilation

[svrnm]

Feature Request

Is your feature request related to a problem?

As stated in this comment by @austinlparker the build for arm64 takes ~4hrs right now. This time hopefully should improve due to the plenty improvements that now exist for the Dockerfiles, but will probably still be significant (e.g. the rust image took 1+hrs and is not at all optimized)

Describe the solution you'd like:

I suggest making use of cross compilation. I did a tiny experiment with the checkoutservice and it improved the build times from ~7mins to ~2mins:

https://github.com/svrnm/playground/actions/runs/3981194659
https://github.com/svrnm/playground/actions/runs/3981102723

The dockerfiles need to be changed for that as outlined in this blog post, e.g. here's my checkout service Dockerfile for cross compilation:

https://github.com/svrnm/playground/blob/main/src/checkoutservice/Dockerfile

The good thing is, that those changes don't need to be introduced at once and also not for all services. My suggestion is that we pick the ones which take the longest:

• Currency
• Payment
• Quote
• Shipping

What do you all think?

[cartersocha]

FYI @open-telemetry/demo-approvers

super interesting and worth pursuing I think

[austinlparker]

I'm not entirely sure this is going to help with the worst-case stuff. It's a nice reduction in time, but since builds happen monolithically (every image is built on release, we don't do builds for specific changed containers during PRs, etc) saving time on go doesn't really help unless we get the really slow stuff broken up and we can't cross-compile the dependency chains for grpc afaik?

[svrnm]

@austinlparker this also works for C++/Rust/..., I just picked Go as an example because it was kind of easy. I played with the rust stuff a little bit, but have no results yet. In theory you should be able to cross-compile everything (including the grpc dependency chain), so the ideal setup is to have a build stage that does 99% of the heavy lifting and a release stage that just copies files out of that

[austinlparker]

Hm, when I tried doing the cross-compilation stuff I wasn't able to get it to work on Rust (and the C++ side was really frustrating), maybe you'll have better luck.

[svrnm]

Hm, when I tried doing the cross-compilation stuff I wasn't able to get it to work on Rust (and the C++ side was really frustrating), maybe you'll have better luck.

I will at least give it a try, if I succeed awesome, if not this issue here will be a reminder of my hubris and a warning for future contributors.

[svrnm]

Quick update on that, I did a trial with the current state of all Dockerfiles & one more with the patched Dockerfile for the shipping-service. Without testing functionality I can report that building for AMD64 & ARM64 takes

• 44 minutes with what we have as of today
• 20 minutes with the cross-compilation for shipping service. With that the featureflag service is taking the longest, currencyservice comes second.

To be honest, I am not sure if adding #715 is worth the hassle, 44min sounds OK-ish to me.

[austinlparker]

Honestly 20 minutes is a lot closer to where we wanted to be... I believe our target is 15m. I agree that the new Dockerfile is pretty ugly, but it shouldn't need to be putzed with a lot, right?

[cartersocha]

A decreased build time is more useful to project contributors imo.

I barely look at the docker files personally but have to build things a lot and the Apple silicon issue isn't going away

+1 on build time over complexity

[svrnm]

Ok, I was not aware of that 15min target ;-)

I agree that the new Dockerfile is pretty ugly, but it shouldn't need to be putzed with a lot, right?

True, and it still can be converted to something more simple if we run into any issues.

I did some reading yesterday on branching in Dockerfiles, so maybe there's an option to tidy up things, will give it another look later this week

[puckpuck]

Building locally for a single architecture is much faster now.
Intel MacBook Pro (2019): 12m45s
M2 Max MacBook (2023): 4m41s

The 1.3.1 release, which is multiple architectures for amd64 and arm64, took 24m23s.

All these build times are reasonable. Should we close this issue?

[svrnm]

looks good enough for me to have this closed