Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(ci): Enable S3 caching for interop #1193

Merged
merged 8 commits into from
Sep 26, 2024
Merged

Conversation

AlejandroCabeza
Copy link
Collaborator

@AlejandroCabeza AlejandroCabeza commented Sep 12, 2024

  • Adds our S3 bucket for caching docker images as Protocol Labs shut down their shared one.
  • Remove the free disk space workaround that prevented the jobs from failing for using too much space for the images.

Closes: #1173

Comment on lines +53 to +56
s3-cache-bucket: ${{ vars.S3_LIBP2P_BUILD_CACHE_BUCKET_NAME }}
s3-access-key-id: ${{ vars.S3_LIBP2P_BUILD_CACHE_AWS_ACCESS_KEY_ID }}
s3-secret-access-key: ${{ secrets.S3_LIBP2P_BUILD_CACHE_AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.S3_LIBP2P_BUILD_CACHE_AWS_REGION }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it be possible to avoid the duplication?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which duplication?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, in both calls, you mean. Not that I know.

Copy link
Contributor

@diegomrsantos diegomrsantos Sep 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

something like this might work

name: Interoperability Tests

on:
  pull_request:
  merge_group:
  push:
    branches:
      - master
  workflow_dispatch:

concurrency:
  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: true

jobs:
  run-interop-tests:
    runs-on: ubuntu-22.04
    strategy:
      matrix:
        test-type: 
          - transport-interop
          - hole-punching-interop

    name: Run ${{ matrix.test-type }} Tests
    
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Setup Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build Docker Image
        run: |
          docker buildx build --load -t nim-libp2p-head -f tests/${{ matrix.test-type }}/Dockerfile .

      - name: Run Tests
        uses: libp2p/test-plans/.github/actions/run-${{ matrix.test-type == 'transport-interop' && 'transport-interop-test' || 'interop-hole-punch-test' }}@master
        with:
          test-filter: nim-libp2p-head
          extra-versions: ${{ github.workspace }}/tests/${{ matrix.test-type }}/version.json
          s3-cache-bucket: ${{ vars.S3_LIBP2P_BUILD_CACHE_BUCKET_NAME }}
          s3-access-key-id: ${{ vars.S3_LIBP2P_BUILD_CACHE_AWS_ACCESS_KEY_ID }}
          s3-secret-access-key: ${{ secrets.S3_LIBP2P_BUILD_CACHE_AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ vars.S3_LIBP2P_BUILD_CACHE_AWS_REGION }}

Copy link
Collaborator Author

@AlejandroCabeza AlejandroCabeza Sep 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally feel it's clearer the other way, so for me the tradeoff is not worth.
If you still want to go with it please confirm and I'll make the change.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kinda agree with @AlejandroCabeza here. Sometimes, trying to remove duplicates at all cost is detrimental to understanding the code.

Copy link
Contributor

@diegomrsantos diegomrsantos Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Particularly, I don't think this code is much harder to understand at all. Both jobs are a duplication of each other and it's not a good practice.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll wait for an agreement on this one.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, it's not a very strong feeling on my part, we can change if Diego is adamant about it.

Copy link
Collaborator

@lchenut lchenut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@diegomrsantos
Copy link
Contributor

I believe this isn't necessary anymore

  - name: Free Disk Space (Ubuntu)
        # For some reason the original job (libp2p/test-plans) has enough disk space, but this one doesn't.
        uses: jlumbroso/free-disk-space@v1.3.1
        with:
          tool-cache: true

runs-on: ubuntu-22.04
if: success() || failure()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing this isn't the purpose of this PR

@diegomrsantos
Copy link
Contributor

Unfortunately, it's not possible to do that as it seems thatuses doesn't support creating its value dynamically. We can rollback those last changes, remove the Free Disk Space step, and live with the duplication for now.

@diegomrsantos
Copy link
Contributor

@lchenut please approve it again as my approval isn't considered.

@diegomrsantos diegomrsantos added this pull request to the merge queue Sep 26, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 26, 2024
@diegomrsantos diegomrsantos added this pull request to the merge queue Sep 26, 2024
Merged via the queue into master with commit 5d48776 Sep 26, 2024
14 checks passed
@diegomrsantos diegomrsantos deleted the feature/interop-caching branch September 26, 2024 10:54
shashankshampi added a commit that referenced this pull request Oct 1, 2024
added test wrt subscribe and unsubscribe

added tests/pubsub/testgossipinternal2 file

linters

feat: rendezvous refactor (#1183)

Hello!

This PR aim to refactor rendezvous code so that it is easier to impl.
Waku rdv strategy. The hardcoded min and max TTL were out of range with
what we needed and specifying which peers to interact with is also
needed since Waku deals with peers on multiple separate shards.

I tried to keep the changes to a minimum, specifically I did not change
the name of any public procs which result in less than descriptive names
in some cases. I also wanted to return results instead of raising
exceptions but didn't. Would it be acceptable to do so?

Please advise on best practices, thank you.

---------

Co-authored-by: Ludovic Chenut <ludovic@status.im>

refactor and suite name refactor

chore(ci): Enable S3 caching for interop (#1193)

- Adds our S3 bucket for caching docker images as Protocol Labs shut
down their shared one.
- Remove the free disk space workaround that prevented the jobs from
failing for using too much space for the images.

---------

Co-authored-by: diegomrsantos <diego@status.im>

PR review comment changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: done
Development

Successfully merging this pull request may close these issues.

Fix interop test cache
3 participants