Investigate unexpected changes to Docker Asset Hashing

[rix0rrr]

In our integration tests & canaries, we are seeing that the same Docker image gets tagged with 1000 different asset hashes.

How it works

Asset hashes are (should be) based on the contents of the Dockerfile directly, whether all files in there are used or not. Example:

Dockerfile
one.txt
two.txt
three.txt   <-- if these files are not referenced in "Dockerfile", they do contribute to the source hash, but not to the actual image hash

The image digest (the unique identifier) is based on the actual built artifact, which depends on what's actually in the Dockerfile.

It can therefore happen that adding four.txt to the directory changes our calculated hash, but doesn't actually change the Docker image that gets built; that way, multiple images can end up with different asset hash tags.

The problem

In the integ tests, we are seeing 100s or 1000 tags on a unique image (the ecr.public/alpine:latest image), even though nothing should have changed between runs. We seem to be adding ~10 tags/day.

Something is affecting the hash that gets calculated on every canary run and integ test run, but we don't know what it is. Investigate that, and see if there's a problem in the construct lib that we need to fix.

[iliapolo]

Adding my investigation. Querying CloudWatch for canary logs clearly shows that the same test, executed at different runs, produces different hashes:

| 2024-11-05 15:01:33.815 | [15:01:32] cdktest-0esm4a6pqx67-docker-with-custom-file: debug: docker push 123456789111.dkr.ecr.us-east-1.amazonaws.com/cdk-hnb659fds-container-assets-123456789111-us-east-1:89f5fb0c93b481d9f93a18385ec124b797b8ba2deb3eb90d455439065ab5c36b
| 2024-11-05 12:56:41.571 | [12:56:39] cdktest-09llrvhtxofv-docker-with-custom-file: debug: docker push 123456789111.dkr.ecr.us-east-1.amazonaws.com/cdk-hnb659fds-container-assets-123456789111-us-east-1:a4801e285f859adfbce021e712a7f51e2e144291fbfa0e630f6e8c9159a7e395

| 2024-11-05 12:52:11.504 | [12:52:11] cdktest-0hvl3i6c1ui-docker: debug: docker push 123456789111.dkr.ecr.us-east-1.amazonaws.com/cdk-hnb659fds-container-assets-123456789111-us-east-1:a7fa53767f02132dba052cbc454bd54cee6fe3d9eaa1388323829c7bc40e3303
| 2024-11-05 08:56:59.778 | [08:56:57] cdktest-0xjyv4v0xi8b-docker: debug: docker push 123456789111.dkr.ecr.us-east-1.amazonaws.com/cdk-hnb659fds-container-assets-123456789111-us-east-1:d614b4dc455bd31b4cdf2bea47da5ddbdaf81a5213b74888cf153a435a0526da

All 4 of these tags end up in the same image, because as @rix0rrr mentioned, there is no impact on the image digest itself.

Some things to consider that are different between executions:

• The directory the test executes in (temp dir)
• The timestamp of the files in the docker app (because we clone the source dir into a new temporary dir every time)

However, I ran these tests locally multiple times and could not reproduce the issue - I see the same 2 hashes between produced on every run.

So, its unclear where we go from here. We know that these two tests are somehow producing different hashes in some circumstances, but we have to yet to identify what those circumstances are.