Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#429] add docker cache to build workflow #440

Merged
merged 1 commit into from
Oct 25, 2024
Merged

[#429] add docker cache to build workflow #440

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 27 additions & 40 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,63 +49,45 @@ jobs:
})
}
console.log("Clear completed")
- name: Login to Docker Hub
uses: docker/login-action@v3
- name: Install dependencies
uses: ./.github/actions/install_dependencies
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Install required packages
run: |
sudo apt-get update
sudo apt install -y build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev curl \
git libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev zip unzip \
libpython3.11
- name: Install Python
uses: gabrielfalcao/pyenv-action@v18
with:
default: 3.11.4
docker-username: ${{ secrets.DOCKERHUB_USERNAME }}
docker-token: ${{ secrets.DOCKERHUB_TOKEN }}
#NB: We restore/save cache manually so that we save the cache even if the build fails
- name: Load m2 repository cache # Manually caching .m2 repo as the setup-java caching isn't falling back to older caches
id: cached-m2-repo
uses: actions/cache@v4
uses: actions/cache/restore@v4
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I: By using the restore granular action, we prevent the automated cache-save logic that doesn't meet our needs of saving even when the build fails. (Prior to this change we were basically always saving these caches twice.)

with:
path: ~/.m2/repository
key: maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
maven-
- name: Load m2 build cache
id: cached-m2-build
uses: actions/cache@v4
uses: actions/cache/restore@v4
with:
path: ~/.m2/build-cache
key: maven-build-cache-${{ hashFiles('**/pom.xml') }}
restore-keys: |
maven-build-cache-
- name: Install Poetry
uses: snok/install-poetry@v1
- name: Load docker build cache
id: cached-docker-build
uses: actions/cache/restore@v4
with:
path: ~/.docker/cache
key: docker-cache-${{ hashFiles('**/Dockerfile') }}
Copy link
Contributor Author

@ewilkins-csi ewilkins-csi Oct 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I: This has some limitations, as it doesn't also include the files that are copied into the docker images (e.g. our python modules) but this should be close enough to the ideal case without a ton of complexity. Alternatively, since our caches are cumulative anyway, we could just use a timestamp everywhere instead of file hashes.

restore-keys: |
docker-cache-
#NB: Not saving poetry cache on failure in case it's a failure caused by an in-flight python package release
- name: Poetry cache
id: cached-poetry
uses: actions/cache@v4
with:
path: ~/.cache/pypoetry
key: poetry-cache-${{ hashFiles('**/pom.xml') }}
key: poetry-cache-${{ hashFiles('**/pyproject.toml') }}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I: Use pyproject.toml so the cache is updated when the python dependencies change, even if the POMs have not changed.

restore-keys: |
poetry-
- name: Install Helm
run: |
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
- name: Install Helm Unittest Plugin
run: |
echo "Updating helm unittest plugin to latest version..."
helm plugin install https://github.com/helm-unittest/helm-unittest.git
- name: Set up JDK 17
uses: actions/setup-java@v4
with:
java-version: '17'
distribution: 'temurin'
- name: Create Docker Builder Config File
run: sudo touch /etc/buildkitd.toml
poetry-cache-
# Generate the settings.xml for ghcr.io, pypi, & dev-pypi server profiles
- name: Create settings.xml
run: |
Expand All @@ -123,19 +105,24 @@ jobs:
- name: Run Archetype Tests
run: |
./mvnw -B clean install -Parchetype-test -pl :foundation-archetype
#NB: The following two explicit cache saves are necessary to ensure caches are saved on build failure,
# until https://github.com/actions/cache/issues/1315 is resolved
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I: This issue is unlikely to be resolved any time soon as there are underlying restrictions that prevent it. Instead, we follow the guidance of the action maintainers.

- name: Save m2 repository cache
id: save-m2-repo
uses: actions/cache/save@v4
if: always()
if: always() && steps.cached-m2-repo.outputs.cache-hit != 'true'
with:
path: ~/.m2/repository
key: maven-${{ hashFiles('**/pom.xml') }}
- name: Save m2 build cache
id: save-m2-build
uses: actions/cache/save@v4
if: always()
if: always() && steps.cached-m2-build.outputs.cache-hit != 'true'
with:
path: ~/.m2/build-cache
key: maven-build-cache-${{ hashFiles('**/pom.xml') }}
- name: Save docker build cache
id: save-docker-build
uses: actions/cache/save@v4
if: always() && steps.cached-docker-build.outputs.cache-hit != 'true'
with:
path: ~/.docker/cache
key: docker-cache-${{ hashFiles('**/Dockerfile') }}