Compress Docker images with zstd

[sarayourfriend]

Problem

ECS FARGATE does not cache images as they are pulled. This means it must re-pull the image from GHCR for each task that is started during a deployment. The time it takes to pull an image from GHCR has a significant impact on the total length of our deployments, especially for the API which deploys tasks one by one rather than simultaneously as it does for the frontend.

Description

Read the documentation for zstd compression here: https://aws.amazon.com/blogs/containers/reducing-aws-fargate-startup-times-with-zstd-compressed-container-images/

Update our docker builds to use zstd compression.

Docker for macOS apparently does not yet support zstd compression. Implementing this issue would remove the ability for maintainers on macOS to run deployed images locally. This could cause issues with debugging. This issue might not be possible to responsibly implement until after Docker for macOS supports zstd.

[dhruvkb]

I looked into it and it seems to be a broader problem than Docker for macOS. Docker, irrespective of the OS, does not seem to support zstd compression.

The following lines from the AWS Fargate write up (linked in the original issue description) also suggest that zstd compression is currently only supported by OCI images and not by the Docker Engine.

However, I think it is possible to continue to use Docker images for all local development (built using just dc build) and push zstd compressed images to GHCR (as a part of the CI + CD workflow).

[sarayourfriend]

Beyond compatibility and long-term support, we should also consider what the potential savings of this are vs other image slimming techniques (like ejecting development dependencies from the deployed image).

[sarayourfriend]

I'm going to close this @dhruvkb, after thinking about this a bit more. The compatibility questions aren't worth it and we aren't seeing the same issues here. The biggest improvement in deployments we would actually benefit from is switching to some kind of multi-stage deployment where a single API task is deployed first, then the rest are spun up simultaneously. We could accomplish this, potentially, by having two API services, one very small one that is only responsible for running migrations etc, never servicing requests, then a separate real one and having they deploy in order. That's just one idea. In any case, there are other ways that wouldn't cause maintainability or compatibility issues by trying. I don't think image compression is going to give us the best result compared to actually reducing the real size of our Docker image, either.