Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Common base layer optimization #220

Merged
merged 154 commits into from
Aug 29, 2023
Merged

Common base layer optimization #220

merged 154 commits into from
Aug 29, 2023

Conversation

ShutdownRepo
Copy link
Member

Description

Before this change, all images had a debian:11-slim base layer, while all images shared a common stack (package_base.sh). After this change, images will have a exegol:base base layer. This base image is common for all release images (ad, osint, full, light, web) so that some space is saved if a user downloads more than one.

Created four new DockerHub registries:

Pipelines are changed accordingly and will now be structured as follows.
Pipelines were also greatly improved in the process.

image

Brainstorm log

Not doing multi stage because our use case doesn't allow it.
Let imageA and imageB share common install packages:

# imageA
FROM alpine
RUN package1.sh
RUN package2.sh
RUN package3.sh
RUN package4.sh
# imageB
FROM alpine
RUN package1.sh
RUN package4.sh

While RUN package4.sh is common for both images, Docker layers are dependent on their n-1 layer.

imageA = layer0 (FROM alpine) + layer1 (diff between RUN package1.sh and layer0) + layer2 (diff between RUN package2.sh and layer1) + layer3 (diff between RUN package3.sh and layer2) + layer4 (diff between RUN package4.sh and layer3)
imageB = layer0 (FROM alpine) + layer1 (diff between RUN package1.sh and layer0) + layer2 (diff between RUN package4.sh and layer1)

In this example, imageA(layer4) ≠ imageB(layer2) even if the same package4.sh is called. We can't just have an exegol image per package, and them "add" them together. Docker doesn't work like this.
We could however build an image for each package and move files around like explained in Docker's multi-stage building docs but it would require a huge refactor of the way we develop and build new images (.sh packages, pipelines, tool install functions, ...). We also think it would negatively impact the whole project as contributing to it would be harder, and we need the project to stay community-centric.

Our conclusion: not doing multi-stage, but instead optimize what we can with having a common base for all images. We deem it to be the right balance between technical optimization and user/developer experience.

resolves #96

@ShutdownRepo ShutdownRepo added the documentation Improvements or additions to documentation label Aug 29, 2023
@ShutdownRepo ShutdownRepo self-assigned this Aug 29, 2023
@ShutdownRepo ShutdownRepo mentioned this pull request Aug 29, 2023
@ShutdownRepo ShutdownRepo merged commit aa205a4 into dev Aug 29, 2023
40 checks passed
@ShutdownRepo ShutdownRepo added enhancement New feature or request and removed documentation Improvements or additions to documentation labels Aug 29, 2023
@ShutdownRepo ShutdownRepo deleted the refacto2 branch August 29, 2023 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant