ci: cache test images for integration, VM and module tests

[DmitriyLewen]

Description

We got a lot of 429 errors for the tests.
To avoid loading the test images on every run, we need to cache them.

Test runs:

• save images in cache - https://github.com/DmitriyLewen/trivy/actions/runs/11050734568
• take cache for tests - https://github.com/DmitriyLewen/trivy/actions/runs/11050825154
• saved cache:
  

Note

This PR doesn't fix all 429 errors.

• We can still get error when requesting tags for trivy-test-images and trivy-test-vm-images (to make sure that fs contains all required image):
  

  trivy/magefiles/fixture.go

  Lines 23 to 32 in 8128ecc

  	tags, err := crane.ListTags(testImages)
  	if err != nil {
  	return err
  	}
  	for _, tag := range tags {
  	fileName := tag + ".tar.gz"
  	filePath := filepath.Join(dir, fileName)
  	if exists(filePath) {
  	continue
  	}

• TestFanal_Library_DockerLessMode uses remote images:
  

  trivy/pkg/fanal/test/integration/library_test.go

  Lines 156 to 159 in ab0fd0d

  	_, _ = cli.ImageRemove(ctx, tt.remoteImageName, dimage.RemoveOptions{
  	Force: true,
  	PruneChildren: true,
  	})

Related issues

• Close ci: cache trivy-test-images #7598

Checklist

• I've read the guidelines for contributing to this repository.
• I've followed the conventions in the PR title.
• I've added tests that prove my fix is effective or that my feature works.
• I've updated the documentation with the relevant information (if needed).
• I've added usage information (if the PR introduces new options)
• I've included a "before" and "after" example to the description (if the PR is a user interface change).

[DmitriyLewen]

@knqyf263 I added trivy-test-images (trivy-test-vm-images) last update date to add the ability to control and influence the cache.
Tell me if you have other ideas.

[knqyf263]

I want to automatically invalidate the cache when a new image is added to trivy-test-images rather than modifying the key manually. How about something like this?

      - name: Generate image list digest
        id: image-digest
        run: |
          IMAGE_LIST=$(skopeo list-tags docker://ghcr.io/aquasecurity/trivy-test-images) # skopeo is pre-installed
          DIGEST=$(echo "$IMAGE_LIST" | sha256sum | cut -d' ' -f1)
          echo "digest=$DIGEST" >> $GITHUB_OUTPUT

      - name: Restore test images from cache
        uses: actions/cache@v4
        id: restore-test-images
        with:
          path: integration/testdata/fixtures/images
          key: cache-test-images-${{ steps.image-digest.outputs.digest }}
          restore-keys:
            cacche-test-images-

      - name: Download missing test images
        if: steps.restore-test-images.outputs.cache-hit != 'true'
        run: mage test:fixtureContainerImages

It doesn't work if the existing images are updated, though.

[knqyf263]

Also, we don't need a step to save cache because it automatically creates a new cache.

On a cache miss, the action automatically creates a new cache if the job completes successfully.

https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#matching-a-cache-key

[knqyf263]

It doesn't work if the existing images are updated, though.

Ideally, we should calculate a cache key from each image manifest, but a list of images should be enough at the moment. We can think about it later.

          REPO="ghcr.io/aquasecurity/trivy-test-images"
          TAGS=$(crane ls $REPO)
          COMBINED_DIGEST=""
          for TAG in $TAGS; do
            DIGEST=$(crane manifest $REPO:$TAG | sha256sum | cut -d' ' -f1)
            COMBINED_DIGEST="${COMBINED_DIGEST}${DIGEST}"
          done
          FINAL_DIGEST=$(echo "$COMBINED_DIGEST" | sha256sum | cut -d' ' -f1)
          echo "digest=$FINAL_DIGEST" >> $GITHUB_OUTPUT

[DmitriyLewen]

Ideally, we should calculate a cache key from each image manifest, but a list of images should be enough at the moment. We can think about it later.

we should definitely do this in the future, because when updating the test image - the digest will remain the same

[knqyf263]

we should definitely do this in the future, because when updating the test image - the digest will remain the same

Exactly. However, since we pin the digest, it is less likely to happen unless we intentionally change the digest.
https://github.com/aquasecurity/trivy-test-images/blob/bcbd1ca7b1a9d001b7f21f41310b8a17bc355dd2/copy-images.sh#L16

[DmitriyLewen]

@knqyf263 Used your logic and removed the save step c015645

[knqyf263]

I want to automatically invalidate the cache when a new image is added to trivy-test-images rather than modifying the key manually. How about something like this?

      - name: Generate image list digest
        id: image-digest
        run: |
          IMAGE_LIST=$(skopeo list-tags docker://ghcr.io/aquasecurity/trivy-test-images) # skopeo is pre-installed
          DIGEST=$(echo "$IMAGE_LIST" | sha256sum | cut -d' ' -f1)
          echo "digest=$DIGEST" >> $GITHUB_OUTPUT

      - name: Restore test images from cache
        uses: actions/cache@v4
        id: restore-test-images
        with:
          path: integration/testdata/fixtures/images
          key: cache-test-images-${{ steps.image-digest.outputs.digest }}
          restore-keys:
            cacche-test-images-

      - name: Download missing test images
        if: steps.restore-test-images.outputs.cache-hit != 'true'
        run: mage test:fixtureContainerImages

It doesn't work if the existing images are updated, though.

[knqyf263]

Also, we don't need a step to save cache because it automatically creates a new cache.

On a cache miss, the action automatically creates a new cache if the job completes successfully.

https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#matching-a-cache-key

[knqyf263]

In short, the problem with the current logic is that the test fails whenever a test image is added. Then, we need to update the cache key manually. I would like to avoid this somehow by leveraging a cache key and restore keys.
https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#cache-hits-and-misses

[knqyf263]

We don't need this step anymore. It is transparently called.

trivy/magefiles/magefile.go

Line 277 in bbc8e1d

mg.Deps(t.FixtureContainerImages)

And it's efficient as it skips images if it exists locally.

trivy/magefiles/fixture.go

Lines 30 to 32 in bbc8e1d

	if exists(filePath) {
	continue
	}

Now, we call this check twice, which is just time-consuming (and API-consuming).

[DmitriyLewen]

Right, since the cache will still be saved at the end of the job!

Removed download steps in 0609e5d

[knqyf263]

I saw an error.

Post job cleanup.
/usr/bin/tar --posix -cf cache.tzst --exclude cache.tzst -P -C /home/runner/work/trivy/trivy --files-from manifest.txt --use-compress-program zstdmt
Failed to save: Unable to reserve cache with key cache-test-images-e57[2](https://github.com/aquasecurity/trivy/actions/runs/11050771503/job/30699240630?pr=7599#step:14:2)ef093aa79559e3e3fee0916b10fa8804dd0a8836d11eaea83e22c4f8047a, another job may be creating this cache. More details: Cache already exists. Scope: refs/pull/7599/merge, Key: cache-test-images-e572ef09[3](https://github.com/aquasecurity/trivy/actions/runs/11050771503/job/30699240630?pr=7599#step:14:3)aa79559e3e3fee0916b10fa8804dd0a8836d11eaea83e22c4f8047a, Version: f079f76b9e5c88a6d1d7a2a554815bd7b73babbae6f49b403add394d403a41c4

https://github.com/aquasecurity/trivy/actions/runs/11050771503/job/30699240630?pr=7599

But it looks like the module integration test was kicked at the same time and already created the cache with the same key. I don't think it's a problem. Just sharing information.
https://github.com/aquasecurity/trivy/actions/runs/11050771503/job/30699240939?pr=7599

[knqyf263]

I confirmed the cache is restored correctly and images are not pulled.

Run actions/cache@v4
Received 0 of 1848896025 (0.0%), 0.0 MBs/sec
Received 75497472 of 1848896025 (4.1%), 36.0 MBs/sec
Received 171966464 of 1848896025 (9.3%), 54.6 MBs/sec
Received 281018368 of 1848896025 (15.2%), 67.0 MBs/sec
Received 385875968 of 1848896025 (20.9%), 73.6 MBs/sec
Received 482344960 of 1848896025 (26.1%), 76.6 MBs/sec
Received 583008256 of 1848896025 (31.5%), 79.4 MBs/sec
Received 683671552 of 1848896025 (37.0%), 81.5 MBs/sec
Received 784334848 of 1848896025 (42.4%), 83.1 MBs/sec
Received 884998144 of 1848896025 (47.9%), 84.4 MBs/sec
Received 985661440 of 1848896025 (53.3%), 85.4 MBs/sec
Received 1090519040 of 1848896025 (59.0%), 86.6 MBs/sec
Received 1178599424 of 1848896025 (63.7%), 86.4 MBs/sec
Received [12](https://github.com/aquasecurity/trivy/actions/runs/11051066331/job/30700170044?pr=7599#step:6:13)79262720 of 1848896025 (69.2%), 87.1 MBs/sec
Received [13](https://github.com/aquasecurity/trivy/actions/runs/11051066331/job/30700170044?pr=7599#step:6:14)75731712 of 1848896025 (74.4%), 87.4 MBs/sec
Received [14](https://github.com/aquasecurity/trivy/actions/runs/11051066331/job/30700170044?pr=7599#step:6:15)55423488 of 1848896025 (78.7%), 86.7 MBs/sec
Received [15](https://github.com/aquasecurity/trivy/actions/runs/11051066331/job/30700170044?pr=7599#step:6:16)51892480 of 1848896025 (83.9%), 87.0 MBs/sec
Received 1652555776 of 1848896025 (89.4%), 87.5 MBs/sec
Received 1753219072 of 1848896025 (94.8%), 88.0 MBs/sec
Received 1844701721 of 1848896025 (99.8%), 87.9 MBs/sec
Cache Size: ~1763 MB (1848896025 B)
/usr/bin/tar -xf /home/runner/work/_temp/46c2dffa-9fd7-4890-9ec7-13d232adc733/cache.tzst -P -C /home/runner/work/trivy/trivy --use-compress-program unzstd
Received 1848896025 of 1848896025 (100.0%), 83.9 MBs/sec
Cache restored successfully
Cache restored from key: cache-test-images-e572ef093aa79559e3e3fee09[16](https://github.com/aquasecurity/trivy/actions/runs/11051066331/job/30700170044?pr=7599#step:6:17)b10fa8804dd0a8836d11eaea83e22c4f8047a

[knqyf263]

nit: We no longer need id.

[knqyf263]

We want to merge this PR soon. We can delete it later.

[knqyf263]

Why is cache missing...?

Cache not found for input keys: cache-test-images-e572ef093aa79559e3e3fee0916b10fa8804dd0a8836d11eaea83e22c4f8047a, cache-test-images-

[knqyf263]

By any chance, does the merge queue not share the cache with the normal workflow?

[knqyf263]

I guess this is relevant. The merge queue creates a dedicated branch, which cannot access the cache in the cache-test-images branch.
https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#restrictions-for-accessing-a-cache

[DmitriyLewen]

Workflow runs can restore caches created in either the current branch or the default branch (usually main)

UUIC Once we cache images in main branch, new branches created by the merge queue will also be able to access that cache (from the main branch), right?

[knqyf263]

new branches created by the merge queue will also be able to access that cache (from the main branch), right?

Yes. We need to successfully pull images in the main branch only once .