Reproducible builds with spring boot: Different image names result in different image IDs

[schnatterer]

Preface
This issue is about building spring boot images but I'm not sure which build pack actually causes my issue, so I'm posting it here in the builder. Please move this issue, if it fits better somewhere else.

Actual issue

• I'm looking for ways to get deterministic builds for spring boot projects using build packs.
• I've worked out so far that my best bet is to pin the builder image's repo digest.
• With a builder image from August 2020 this works as expected. That is, whenever I build the same code, the resulting image ID is the same.
• With the latest builder image this no longer works. That is, building the same code results in a new image ID when different image names are set.

Here's an example:

# Just get some project to build
cd /tmp
wget -O demo.zip https://start.spring.io/starter.zip?type=maven-project&language=java&bootVersion=2.3.5.RELEASE&baseDir=demo&groupId=com.example&artifactId=demo&name=demo&description=Demo%20project%20for%20Spring%20Boot&packageName=com.example.demo&packaging=jar&javaVersion=11
unzip demo.zip
cd demo

# Unexpected: Building the same code but using different image names (created with the latest builder image, uploaded November 7 2020): Different image IDs
BUILDER=gcr.io/paketo-buildpacks/builder@sha256:bae0a6b7ec5d51867eb49ac598775bb5fa8788bba458d2edfda314611be554a8
./mvnw clean spring-boot:build-image -DskipTests -Dspring-boot.build-image.imageName=$(date +%s) -Dspring-boot.build-image.builder=$BUILDER | grep -A1 '*** Images'
./mvnw clean spring-boot:build-image -DskipTests -Dspring-boot.build-image.imageName=$(date +%s) -Dspring-boot.build-image.builder=$BUILDER | grep -A1 '*** Images'

[INFO]     [creator]     *** Images (b20d54746b9a):
[INFO]     [creator]           docker.io/library/1604994725:latest
[INFO]     [creator]     *** Images (1992ec6cbc31):
[INFO]     [creator]           docker.io/library/1604994755:latest

# Same image, when image name is the same
./mvnw clean spring-boot:build-image -DskipTests -Dspring-boot.build-image.imageName=1 -Dspring-boot.build-image.builder=$BUILDER | grep -A1 '*** Images'
./mvnw clean spring-boot:build-image -DskipTests -Dspring-boot.build-image.imageName=1 -Dspring-boot.build-image.builder=$BUILDER | grep -A1 '*** Images'

[INFO]     [creator]     *** Images (3c57885e2200):
[INFO]     [creator]           docker.io/library/1:latest
[INFO]     [creator]     *** Images (3c57885e2200):
[INFO]     [creator]           docker.io/library/1:latest

So my workaround would be to use the same image name for an application on each build. But this has a downside: When building multiple branches of the same application on the same machine (e.g. CI server) at the same time, they might overwrite the image name resulting in the wrong images being pushed.

Also

• the imageName AKA docker tag is only a tag. One image can have multiple tags. The image is uniquely identified by it's content addressable ID, not by its tags. So for me it's unexpected that the image tag changes the ID of the image
• this used to work as expected in the past. Same commands, but different builder image (from august 2020)

BUILDER=gcr.io/paketo-buildpacks/builder@sha256:4d77f1f574148ff8f6fcd6ada7bf16559aa9ab080c30cdfc6613cdb235c0fb3b
./mvnw clean spring-boot:build-image -DskipTests -Dspring-boot.build-image.imageName=$(date +%s) -Dspring-boot.build-image.builder=$BUILDER | grep -A1 '*** Images'
./mvnw clean spring-boot:build-image -DskipTests -Dspring-boot.build-image.imageName=$(date +%s) -Dspring-boot.build-image.builder=$BUILDER | grep -A1 '*** Images'
[INFO]     [creator]     *** Images (c0707e59e6ca):
[INFO]     [creator]           docker.io/library/1604994412:latest
[INFO]     [creator]     *** Images (c0707e59e6ca):
[INFO]     [creator]           docker.io/library/1604994465:latest

Any ideas which change (between august to november) caused this change in behavior?

[fg-j]

cc @ekcasey and @nebhale given that this is seemingly an issue coupled with Spring Boot.

[nebhale]

@schnatterer My gut tells me that this is an issue with the Maven plugin in what it passes to the lifecycle during build, but we’ll do the legwork to diagnose it this morning.

/cc @scottfrederick @wilkinsona

[scottfrederick]

this used to work as expected in the past. Same commands, but different builder image (from august 2020)

@schnatterer What versions of the Spring Boot maven plugin were you using before (when this used to work as expected) and after (when you started seeing the issue)?

[schnatterer]

@scottfrederick same Spring Boot maven plugin version, only the builder images are different.
You should be able to reproduce the issue with the examples listed above (using spring boot 2.3.5).

[nebhale]

Nope, it's on us. Go is enumerating files non-deterministically (sound familiar Boot team? 😜) and who the hell knows why the image name would be the trigger for this. Look for fix issues.

[nebhale]

Another part of the solution: pavlo-v-chernykh/keystore-go#23

[schnatterer]

Thank you all for your support.

I just recognized that I forgot to add the repo digest of the builder image from august (the one that used to work) in the issue description 🙈

I added it to the original question. And here's the whole thing again as one script for easier reproduction:

# Just get some project to build
cd /tmp
wget -O demo.zip https://start.spring.io/starter.zip?type=maven-project&language=java&bootVersion=2.3.5.RELEASE&baseDir=demo&groupId=com.example&artifactId=demo&name=demo&description=Demo%20project%20for%20Spring%20Boot&packageName=com.example.demo&packaging=jar&javaVersion=11
unzip demo.zip
cd demo

# Unexpected: Building the same code but using different image names (created with the latest builder image, uploaded November 7 2020): Different image IDs
BUILDER=gcr.io/paketo-buildpacks/builder@sha256:bae0a6b7ec5d51867eb49ac598775bb5fa8788bba458d2edfda314611be554a8
./mvnw clean spring-boot:build-image -DskipTests -Dspring-boot.build-image.imageName=$(date +%s) -Dspring-boot.build-image.builder=$BUILDER | grep -A1 '*** Images'
./mvnw clean spring-boot:build-image -DskipTests -Dspring-boot.build-image.imageName=$(date +%s) -Dspring-boot.build-image.builder=$BUILDER | grep -A1 '*** Images'
echo

# Same image ID, when image name is the same
./mvnw clean spring-boot:build-image -DskipTests -Dspring-boot.build-image.imageName=1 -Dspring-boot.build-image.builder=$BUILDER | grep -A1 '*** Images'
./mvnw clean spring-boot:build-image -DskipTests -Dspring-boot.build-image.imageName=1 -Dspring-boot.build-image.builder=$BUILDER | grep -A1 '*** Images'
echo

# It used to work: Building the same code but using different image names (created with a builder image, uploaded Augst 18 2020): Same image IDs
BUILDER=gcr.io/paketo-buildpacks/builder@sha256:4d77f1f574148ff8f6fcd6ada7bf16559aa9ab080c30cdfc6613cdb235c0fb3b
./mvnw clean spring-boot:build-image -DskipTests -Dspring-boot.build-image.imageName=$(date +%s) -Dspring-boot.build-image.builder=$BUILDER | grep -A1 '*** Images'
./mvnw clean spring-boot:build-image -DskipTests -Dspring-boot.build-image.imageName=$(date +%s) -Dspring-boot.build-image.builder=$BUILDER | grep -A1 '*** Images'

This generates the following output

[INFO]     [creator]     *** Images (cfb9c19bc136):
[INFO]     [creator]           docker.io/library/1605086009:latest
[INFO]     [creator]     *** Images (05548a334b15):
[INFO]     [creator]           docker.io/library/1605086041:latest

[INFO]     [creator]     *** Images (3c57885e2200):
[INFO]     [creator]           docker.io/library/1:latest
[INFO]     [creator]     *** Images (3c57885e2200):
[INFO]     [creator]           docker.io/library/1:latest

[INFO]     [creator]     *** Images (c0707e59e6ca):
[INFO]     [creator]           docker.io/library/1605086089:latest
[INFO]     [creator]     *** Images (c0707e59e6ca):
[INFO]     [creator]           docker.io/library/1605086118:latest

• The first two images are the ones that are unexpected to me (different IDs due to different names)
• Images three and four have the same ID, because they have the same name
• The last two are build with a builder image from august and have the same ID even though the names are different.

[nebhale]

Thanks for the update @schnatterer. We’ve identified the problems, and while it’ll take a little while to get it all through the system (particularly the external contribution), I’ve already verified that we’re making reproducible images with the changes.

[schnatterer]

Thanks very much for the fixes!
Can you estimate when they will be available via the gcr.io/paketo-buildpacks/builder:base-platform-api-0.3 image?
The one published yesterday still doesn't yield reproducible builds:

BUILDER=gcr.io/paketo-buildpacks/builder@sha256:9de1015aae69197446da778eed2236327c71e71f2451d8e939a1a6fef042dd2f
./mvnw clean spring-boot:build-image -DskipTests -Dspring-boot.build-image.imageName=$(date +%s) -Dspring-boot.build-image.builder=$BUILDER | grep -A1 '*** Images'
./mvnw clean spring-boot:build-image -DskipTests -Dspring-boot.build-image.imageName=$(date +%s) -Dspring-boot.build-image.builder=$BUILDER | grep -A1 '*** Images'

[INFO]     [creator]     *** Images (d9b8d9b4feff):
[INFO]     [creator]           docker.io/library/1605545655:latest
[INFO]     [creator]     *** Images (8e72c3313dea):
[INFO]     [creator]           docker.io/library/1605545690:latest

[nebhale]

@schnatterer I’ve now done the releases

• libjvm 1.24.2
• bellsoft-liberica 5.2.1
• java 4.3.0

and while I haven’t verified it directly, I believe base 0.1.16` is now available with your fixes in it.

[schnatterer]

Thanks @nebhale.
I can confirm that my issue is resolved in gcr.io/paketo-buildpacks/builder:0.1.17-base.

BTW are the tags like 0.1.17-base a good choice for reproducible builds (i.e. are they pushed only once) or is it better to stick with repo digests?

#BUILDER=gcr.io/paketo-buildpacks/builder:0.1.17-base
BUILDER=gcr.io/paketo-buildpacks/builder@sha256:aceb25edf4eb74e8040ca278ed5629a43c7cfe2024bd9fe9ad1a88004dcea80f
 ./mvnw clean spring-boot:build-image -DskipTests -Dspring-boot.build-image.imageName=$(date +%s) | grep -A1 '*** Images' 
./mvnw clean spring-boot:build-image -DskipTests -Dspring-boot.build-image.imageName=$(date +%s)  | grep -A1 '*** Images'

[INFO]     [creator]     *** Images (aba7670e8339):
[INFO]     [creator]           docker.io/library/1605610105:latest
[INFO]     [creator]     *** Images (aba7670e8339):
[INFO]     [creator]           docker.io/library/1605610163:latest
``

[nebhale]

@schnatterer I guess it depends on trust really. We don't ever overwrite tags, but Docker doesn't actively prevent (or notify when it does happen) like Git does. So if you trust us, tag names are a much easier way for us to describe when changes are propagated. But if you don't trust us, then hashes are the only true truth in registries. We're sympathetic to both views which is why our release notes include both.

[schnatterer]

@nebhale thanks for elaborating.
I agree, tag names are just much easier to maintain, which is exactly why I ask.
In general, I'm willing to trust, if I know that tags are generally not overwritten. Some tags, like base-platform-api-0.3 are overwritten, though. So I thought I better ask before having to find out the hard way.
Thanks for your ongoing support and until next time 😉