Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Build action should re-use the docker build cache #429

Closed
2 tasks done
ewilkins-csi opened this issue Oct 23, 2024 · 2 comments · Fixed by #440
Closed
2 tasks done

Feature: Build action should re-use the docker build cache #429

ewilkins-csi opened this issue Oct 23, 2024 · 2 comments · Fixed by #440
Assignees
Labels
enhancement New feature or request

Comments

@ewilkins-csi
Copy link
Contributor

ewilkins-csi commented Oct 23, 2024

Description

We save the m2 repository, the maven build cache, and the poetry cache to GitHub caches so we can re-use them across builds. We should also do this with the docker build cache at ~/.docker/cache. Note that we are already consuming over the 10GB limit for GH caches. The docker build cache is expected to be over 5GB and so could cause issues with our caches being purged too aggressively. There will be a feature early next year to allow open source projects to pay for increased cache limits. Further, there is an action available that is a one-to-one replacement for the official cache action that allows writing to S3.

However, on further discussion, we realized that our current pattern of segregating the maven and poetry caches based on the contents of the projects POM files is unnecessary and causing a lot of excess cache usage. If we remove this segregation and just use a constant cache, our cache usage should drop to 6GB. This may be enough that we can get by floating right at the limit. So for the first pass at this, we'll just trim back our other cache usage and see if it becomes an issue. If it does, we can explore the options listed above.

DOD

Acceptance criteria required to realize the requested feature

  • the build.yaml workflow is updated to create/use a docker-cache GH cache to prevent full docker image rebuilds
  • the build.yaml workflow is updated to save the m2 repo, maven build cache, and poetry cache to a constant cache name
    • This is not possible, as GH actions does not allow a cache key to be updated
  • the build.yaml workflow is updated to use the install-dependencies reusable action to install necessary build tools

Test Strategy/Script

  1. Examine the logs for the most recent build
  2. Verify there are stages for restoring the Docker cache and saving the Docker cache
  3. Verify the "Install dependencies" action is used instead of individual steps for installing Java/Python/Helm/etc.
  4. Ensure a cache named docker-cache-* is in GH caches
@ewilkins-csi ewilkins-csi added the enhancement New feature or request label Oct 23, 2024
ewilkins-csi added a commit that referenced this issue Oct 23, 2024
Saves the docker cache configured in the `cacheTo` and `cacheFrom`
options of the Fabric8 `docker-maven-plugin` to GH caches to speed image
rebuilds.  Also uses the `install-dependencies` custom action to
de-duplicate the required build tool installation logic, and updates the
existing caches to save to a constant name.
ewilkins-csi added a commit that referenced this issue Oct 23, 2024
Saves the docker cache configured in the `cacheTo` and `cacheFrom`
options of the Fabric8 `docker-maven-plugin` to GH caches to speed image
rebuilds.  Also uses the `install-dependencies` custom action to
de-duplicate the required build tool installation logic, and updates the
existing caches to save to a constant name.
@ewilkins-csi ewilkins-csi self-assigned this Oct 24, 2024
ewilkins-csi added a commit that referenced this issue Oct 24, 2024
Saves the docker cache configured in the `cacheTo` and `cacheFrom`
options of the Fabric8 `docker-maven-plugin` to GH caches to speed image
rebuilds.  Also uses the `install-dependencies` custom action to
de-duplicate the required build tool installation logic, and updates the
existing caches to save to a constant name.
ewilkins-csi added a commit that referenced this issue Oct 24, 2024
Saves the docker cache configured in the `cacheTo` and `cacheFrom`
options of the Fabric8 `docker-maven-plugin` to GH caches to speed image
rebuilds.  Also uses the `install-dependencies` custom action to
de-duplicate the required build tool installation logic, and updates the
existing caches to save to a constant name.
ewilkins-csi added a commit that referenced this issue Oct 24, 2024
Saves the docker cache configured in the `cacheTo` and `cacheFrom`
options of the Fabric8 `docker-maven-plugin` to GH caches to speed image
rebuilds.  Also uses the `install-dependencies` custom action to
de-duplicate the required build tool installation logic, and updates the
existing caches to save to a constant name.
ewilkins-csi added a commit that referenced this issue Oct 24, 2024
Saves the docker cache configured in the `cacheTo` and `cacheFrom`
options of the Fabric8 `docker-maven-plugin` to GH caches to speed image
rebuilds.  Also uses the `install-dependencies` custom action to
de-duplicate the required build tool installation logic, and updates the
existing caches to save to a constant name.
ewilkins-csi added a commit that referenced this issue Oct 24, 2024
Saves the docker cache configured in the `cacheTo` and `cacheFrom`
options of the Fabric8 `docker-maven-plugin` to GH caches to speed image
rebuilds.  Also uses the `install-dependencies` custom action to
de-duplicate the required build tool installation logic, and updates the
existing caches to save to a constant name.
ewilkins-csi added a commit that referenced this issue Oct 25, 2024
Saves the docker cache configured in the `cacheTo` and `cacheFrom`
options of the Fabric8 `docker-maven-plugin` to GH caches to speed image
rebuilds.  Also uses the `install-dependencies` custom action to
de-duplicate the required build tool installation logic, and updates the
existing caches to save to a constant name.
ewilkins-csi added a commit that referenced this issue Oct 25, 2024
Saves the docker cache configured in the `cacheTo` and `cacheFrom`
options of the Fabric8 `docker-maven-plugin` to GH caches to speed image
rebuilds.  Also uses the `install-dependencies` custom action to
de-duplicate the required build tool installation logic.
ewilkins-csi added a commit that referenced this issue Oct 25, 2024
Saves the docker cache configured in the `cacheTo` and `cacheFrom`
options of the Fabric8 `docker-maven-plugin` to GH caches to speed image
rebuilds.  Also uses the `install-dependencies` custom action to
de-duplicate the required build tool installation logic.
@ewilkins-csi
Copy link
Contributor Author

DOD/OTS with @csun-cpointe

ewilkins-csi added a commit that referenced this issue Oct 25, 2024
@ewilkins-csi ewilkins-csi linked a pull request Oct 25, 2024 that will close this issue
@carter-cundiff
Copy link
Contributor

Testing passed:
image
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants