Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

breaking!(buildDockerAndPublishImage): support multi-platform with Docker Bake for building Linux container images #730

Merged
merged 70 commits into from
Aug 31, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
5455c81
add the bake version for build and deploy
smerle33 Aug 17, 2023
c5552a9
add the dockerbake flag
smerle33 Aug 17, 2023
a76cf12
docker bake file
smerle33 Aug 18, 2023
374d6f8
no load
smerle33 Aug 18, 2023
4dab2ef
fix path for bakefile
smerle33 Aug 18, 2023
120f2bd
correct path for bake file
smerle33 Aug 18, 2023
ec41bd1
DOCKERBAKEFILE env
smerle33 Aug 18, 2023
edca19f
debug path
smerle33 Aug 18, 2023
b6f9246
try to move the container run for qemu
smerle33 Aug 18, 2023
26c5355
build and load
smerle33 Aug 18, 2023
88d7929
remove qemu for test and change simple quotes for load platform
smerle33 Aug 18, 2023
5d475c5
move to env variable the dpkg
smerle33 Aug 18, 2023
c11cbec
add a tag for debug purposes
smerle33 Aug 18, 2023
f0ed588
add a tag for debug purposes
smerle33 Aug 18, 2023
7b39dd0
test tagname as tag
smerle33 Aug 18, 2023
71958d8
tagname
smerle33 Aug 18, 2023
5f56408
change tag to remove beta
smerle33 Aug 18, 2023
48752d9
add some bake debug for tag
smerle33 Aug 18, 2023
2bdca24
add env debug
smerle33 Aug 21, 2023
4951fee
change debug tag version
smerle33 Aug 21, 2023
0d926bb
add some debug for TAG_NAME
smerle33 Aug 21, 2023
e39e83b
more tag name debug
smerle33 Aug 21, 2023
6566e60
revert all debug
smerle33 Aug 21, 2023
e34a354
remove forced env TAG NAME
smerle33 Aug 21, 2023
898c6e1
generated docker bake file
smerle33 Aug 21, 2023
5862c15
template for docker bake
smerle33 Aug 21, 2023
dbbd9c3
rename docker bake file
smerle33 Aug 21, 2023
56d3187
override bake file in variable
smerle33 Aug 21, 2023
6a9466a
wip
smerle33 Aug 21, 2023
a33f95d
wip
smerle33 Aug 21, 2023
77d6b9b
wip
smerle33 Aug 21, 2023
4e7c5f4
debug path to file
smerle33 Aug 21, 2023
072da49
variable as file name
smerle33 Aug 21, 2023
5d6aaee
remove debug and change platforms to array
smerle33 Aug 21, 2023
008a897
list of string
smerle33 Aug 21, 2023
891abc9
list of string
smerle33 Aug 21, 2023
218e5c3
list of string
smerle33 Aug 21, 2023
63295f5
remove debug and change linux flag
smerle33 Aug 22, 2023
bff20b0
add deploy with docker bake override file and still windows deploy
smerle33 Aug 22, 2023
1ad2172
spotless
smerle33 Aug 22, 2023
366f006
add a multiplatform test
smerle33 Aug 22, 2023
7f46ace
take care of the Dockerfile name and context IMAGE_DIR variables
smerle33 Aug 22, 2023
d78bbdc
add debug
smerle33 Aug 22, 2023
f62a347
remove quotes
smerle33 Aug 22, 2023
1cb3f11
remove quotes
smerle33 Aug 22, 2023
19f6f8d
change platform to platforms by defaut
smerle33 Aug 23, 2023
c3768eb
change test for S IMAGE_PLATFORMS
smerle33 Aug 23, 2023
00dbaec
better handle of platform.s
smerle33 Aug 23, 2023
7986a8f
remove env debug
smerle33 Aug 23, 2023
2668f46
remove variable, set directly in command
smerle33 Aug 24, 2023
5c3ce45
reword and restructure after pair programming with damien
smerle33 Aug 24, 2023
b6811f9
mvn spotless:apply
smerle33 Aug 25, 2023
5db7dcf
rename platforms to targetplatforms and change default value else
smerle33 Aug 25, 2023
8e57caf
factorize and chores
smerle33 Aug 25, 2023
b49e110
change test but error on same for build and deploy need to find a wa…
smerle33 Aug 25, 2023
7694831
revert test with correct action
smerle33 Aug 28, 2023
4bfd3b2
replace single quotes by double quotes to enable groovy interpolation
smerle33 Aug 28, 2023
d27e0cc
move tag manually add to the function
smerle33 Aug 29, 2023
edbc5b8
remove duplicate variable definition
smerle33 Aug 29, 2023
a59fb6b
rename variable
smerle33 Aug 29, 2023
7bf6023
add tag only with legacy process for windows target / agent
smerle33 Aug 29, 2023
e929bef
cleanup unit tests
smerle33 Aug 29, 2023
3d02040
cleanup factorisation with damien
smerle33 Aug 29, 2023
a0be4e7
add warning if no target platform specified
smerle33 Aug 29, 2023
7e9a091
add readme and txt documentation
smerle33 Aug 30, 2023
b5f8a0e
add test and unfactorize some
smerle33 Aug 30, 2023
b519834
move some command and remove comments, also add the bake-test to ensu…
smerle33 Aug 30, 2023
aef6007
chore some infos with new comments
smerle33 Aug 30, 2023
e0c105f
take care of comments in PR
smerle33 Aug 30, 2023
fcc0873
using a name builx environement to avoid rebuild on each call
smerle33 Aug 30, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 33 additions & 7 deletions README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -189,18 +189,45 @@ Supported parameters:
`imageName`::
Name of the docker image to build

`configs`::
(Optional) extra flags

registry: override the smart default of jenkinsciinfra/ or jenkins4eval/
dockerfile: override the default dockerfile of Dockerfile
`config`::
(Optional) map of extra flags

* agentLabels: String expression for the labels the agent must match
* automaticSemanticVersioning: Do not automagically increase semantic version by default
* includeImageNameInTag: Set to true for multiple semversioned images built in parallel, will include the image name in tag to avoid conflict
* dockerfile: override the default dockerfile of Dockerfile
* targetplatforms: defined the platforms to build as TARGET
* nextVersionCommand: Commmand line used to retrieve the next version (default 'jx-release-version')
* gitCredentials: override Credential ID for tagging and creating release
* imageDir: Relative path to the context directory for the Docker build
* registryNamespace: empty = autodiscover based on the current controller, but can override the smart default of jenkinsciinfra/ or jenkins4eval/
* unstash: Allow to unstash files if not empty
* dockerBakeFile: Allow to build from a bake file instead

==== Example
[source, groovy]
----
buildDockerImage_k8s('plugins-site-api')
buildDockerAndPublishImage('plugins-site-api')
buildDockerAndPublishImage('inbound-agent-maven:jdk8-nanoserver', [
dockerfile: 'maven/jdk8/Dockerfile.nanoserver',
agentLabels: 'docker-windows-2019 && amd64',
targetplatforms: 'windows/amd64',
imageDir: 'maven/jdk8',
])
----

is also called from `parallelDockerUpdatecli` with `config` within `buildDockerConfig` like this :
[source, groovy]
----
parallelDockerUpdatecli([
imageName: 'wiki',
rebuildImageOnPeriodicJob: false,
updatecliConfig: [containerMemory: '1G'],
buildDockerConfig : [targetplatforms: 'linux/amd64,linux/arm64,linux/s390x']
])
----


== Contribute

=== Requirements
Expand All @@ -213,4 +240,3 @@ buildDockerImage_k8s('plugins-site-api')
By adding `@Library('pipeline-library@pull/<your-pr-number>/head') _` at the top of a Jenkinsfile from a repository built on one of the *.ci.jenkins.io instances, you can test your pipeline library pull request on ci.jenkins.io.

A repository is dedicated for these kind of tests: https://github.com/jenkinsci/jenkins-infra-test-plugin/

21 changes: 19 additions & 2 deletions resources/io/jenkins/infra/docker/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,12 @@ endif

IMAGE_NAME ?= helloworld
IMAGE_DEPLOY_NAME ?= "$(IMAGE_NAME)"
IMAGE_PLATFORM ?= linux/amd64
BUILD_TARGETPLATFORM ?= linux/amd64
dduportal marked this conversation as resolved.
Show resolved Hide resolved
# Paths
IMAGE_DOCKERFILE ?= "$(IMAGE_DIR)"/Dockerfile
HADOLINT_REPORT ?= "$(IMAGE_DIR)"/hadolint.json
TEST_HARNESS ?= "$(IMAGE_DIR)"/cst.yml
DOCKER_BAKE_FILE ?= "$(IMAGE_DIR)"/docker-bake.hcl

## Image metadatas
GIT_COMMIT_REV ?= $(shell git log -n 1 --pretty=format:'%h')
Expand Down Expand Up @@ -47,11 +48,16 @@ build: ## Build the Docker Image $(IMAGE_NAME) from $(IMAGE_DOCKERFILE)
--label "org.label-schema.vcs-ref=$(GIT_COMMIT_REV)" \
--label "org.opencontainers.image.created=$(BUILD_DATE)" \
--label "org.label-schema.build-date=$(BUILD_DATE)" \
--platform "$(IMAGE_PLATFORM)" \
--platform "$(BUILD_TARGETPLATFORM)" \
--file "$(call FixPath,$(IMAGE_DOCKERFILE))" \
"$(IMAGE_DIR)"
@echo "== Build succeeded"

bake-build: ## Build the Docker Image(s) with dockerbake file
@echo "== Building from DockerBake file"
@docker buildx bake -f "$(call FixPath,$(DOCKER_BAKE_FILE))"


clean: ## Delete any file generated during the build steps
@echo "== Cleaning working directory $(IMAGE_DIR) from generated artefacts:"
rm -f "$(call FixPath,$(IMAGE_DIR)/*.tar)" "$(HADOLINT_REPORT)"
Expand All @@ -62,11 +68,22 @@ test: ## Execute the test harness on the Docker Image
container-structure-test test --driver=docker --image="$(IMAGE_NAME)" --config="$(call FixPath,$(TEST_HARNESS))"
@echo "== Test succeeded"

bake-test: ## Execute the test harness on the Docker Image with load
@echo "== Load $(IMAGE_NAME) within docker engine from docker bake buildx engine"
@docker buildx bake -f "$(call FixPath,$(DOCKER_BAKE_FILE))" --set "*.platform=linux/$(shell dpkg --print-architecture)" --load
@echo "== Test $(IMAGE_NAME) with $(call FixPath,$(TEST_HARNESS)) from $(IMAGE_NAME) with container-structure-test:"
container-structure-test test --driver=docker --image="$(IMAGE_NAME)" --config="$(call FixPath,$(TEST_HARNESS))"
@echo "== Test succeeded"

## This steps expects that you are logged to the Docker registry to push image into
deploy: ## Tag and push the built image as specified by $(IMAGE_DEPLOY).
@echo "== Deploying $(IMAGE_NAME) to $(IMAGE_DEPLOY_NAME) with docker:"
docker image tag "$(IMAGE_NAME)" "$(IMAGE_DEPLOY_NAME)"
docker image push "$(IMAGE_DEPLOY_NAME)"
@echo "== Deploy succeeded"

bake-deploy: ## Tag and push the built image as specified by docker bake file
@echo "== Deploying with docker bake file"
@docker buildx bake -f "$(call FixPath,$(DOCKER_BAKE_FILE))" --push

.PHONY: all clean lint build test deploy
52 changes: 52 additions & 0 deletions resources/io/jenkins/infra/docker/jenkinsinfrabakefile.hcl
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
variable "IMAGE_DEPLOY_NAME" {}

variable "REGISTRY" {
default = "docker.io"
}

variable "TAG_NAME" {
default = ""
}

variable "BAKE_TARGETPLATFORMS" {
default = "linux/arm64"
}

variable "IMAGE_DOCKERFILE" {
default = "Dockerfile"
}

variable "IMAGE_DIR" {
default = "."
}

# return the full image name
function "full_image_name" {
params = [tag]
result = notequal("", tag) ? "${REGISTRY}/${IMAGE_DEPLOY_NAME}:${tag}" : "${REGISTRY}/${IMAGE_DEPLOY_NAME}:latest"
}

target "default" {
dockerfile = IMAGE_DOCKERFILE
context = IMAGE_DIR
tags = [
full_image_name("latest"),
full_image_name(TAG_NAME)
]
platforms = [BAKE_TARGETPLATFORMS]
args = {
GIT_COMMIT_REV="$(GIT_COMMIT_REV)",
GIT_SCM_URL="$(GIT_SCM_URL)",
BUILD_DATE="$(BUILD_DATE)",
}
labels = {
"org.opencontainers.image.source"="$(GIT_SCM_URL)",
"org.label-schema.vcs-url"="$(GIT_SCM_URL)",
"org.opencontainers.image.url"="$(SCM_URI)",
"org.label-schema.url"="$(SCM_URI)",
"org.opencontainers.image.revision"="$(GIT_COMMIT_REV)",
"org.label-schema.vcs-ref"="$(GIT_COMMIT_REV)",
"org.opencontainers.image.created"="$(BUILD_DATE)",
"org.label-schema.build-date"="$(BUILD_DATE)",
}
}
Loading