#395 Upgrade IT infrastructure for JDK 17

Update the app naming convention for spark-inf to match the standard chart name default. Update it test dockerfile to use jdk17 Update it helm Tiltfile to be up to date with recent V2 chart changes Correct IT jar shading and copy Added migration to update test docker JDK
boozallen · Oct 10, 2024 · 7eecfee · 7eecfee
1 parent 0a0c902
commit 7eecfee
Show file tree

Hide file tree

Showing 38 changed files with 295 additions and 206 deletions.
diff --git a/DRAFT_RELEASE_NOTES.md b/DRAFT_RELEASE_NOTES.md
@@ -26,13 +26,14 @@ The following steps will upgrade your project to 1.10. These instructions consis
 ## Automatic Upgrades
 To reduce burden of upgrading aiSSEMBLE, the Baton project is used to automate the migration of some files to the new version.  These migrations run automatically when you build your project, and are included by default when you update the `build-parent` version in your root POM.  Below is a description of all of the Baton migrations that are included with this version of aiSSEMBLE.
 
-| Migration Name                                             | Description                                                                                                                                                                                                                                      |
-|------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| upgrade-tiltfile-aissemble-version-migration               | Updates the aiSSEMBLE version within your project's Tiltfile                                                                                        |
-| upgrade-v2-chart-files-aissemble-version-migration         | Updates the Helm chart dependencies within your project's deployment resources (`<YOUR_PROJECT>-deploy/src/main/resources/apps/`) to use the latest version of the aiSSEMBLE   |
-| upgrade-v1-chart-files-aissemble-version-migration         | Updates the docker image tags within your project's deployment resources (`<YOUR_PROJECT>-deploy/src/main/resources/apps/`) to use the latest version of the aiSSEMBLE   |
-| spark-version-upgrade-migration                            | Updates the Spark Application executor failure parameters to their new key name to ensure compatibility with spark `3.5`                                                                  |
-| log4j-maven-shade-plugin-migration                         | Updates the Maven Shade Plugin with the new Log4j dependency information                                                                                     |
+| Migration Name                                     | Description                                                                                                                                                                  |
+|----------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| upgrade-tiltfile-aissemble-version-migration       | Updates the aiSSEMBLE version within your project's Tiltfile                                                                                                                 |
+| upgrade-v2-chart-files-aissemble-version-migration | Updates the Helm chart dependencies within your project's deployment resources (`<YOUR_PROJECT>-deploy/src/main/resources/apps/`) to use the latest version of the aiSSEMBLE |
+| upgrade-v1-chart-files-aissemble-version-migration | Updates the docker image tags within your project's deployment resources (`<YOUR_PROJECT>-deploy/src/main/resources/apps/`) to use the latest version of the aiSSEMBLE       |
+| spark-version-upgrade-migration                    | Updates the Spark Application executor failure parameters to their new key name to ensure compatibility with spark `3.5`                                                     |
+| it-infrastructure-java-upgrade-migration           | Updates the Java docker image version in the integration test docker module to JDK 17                                                                                        |
+| log4j-maven-shade-plugin-migration                 | Updates the Maven Shade Plugin with the new Log4j dependency information                                                                                                     |
 
 To deactivate any of these migrations, add the following configuration to the `baton-maven-plugin` within your root `pom.xml`:
 
@@ -70,6 +71,12 @@ To start your aiSSEMBLE upgrade, update your project's pom.xml to use the 1.10.0
 
 ## Conditional Steps
 
+### Spark Infrastructure App naming
+The Spark Infrastructure helm charts were updated to use `Values.app.name` as the deployment and service name where 
+previously it was `Release.name`. This change applies to the charts `aissemble-spark-history-chart` and  
+`aissemble-thrift-server-chart`. To be compatible with this change you will need to add the `Values.app.name` to 
+these two chart values.
+
 ## Final Steps - Required for All Projects
 ### Finalizing the Upgrade
 1. Run `./mvnw org.technologybrewery.baton:baton-maven-plugin:baton-migrate` to apply the automatic migrations
@@ -78,4 +85,4 @@ To start your aiSSEMBLE upgrade, update your project's pom.xml to use the 1.10.0
 3. Repeat the previous step until all manual actions are resolved
 
 # What's Changed
-- `pyproject.toml` files updated to allow for Python version `>=3.8`.
+- `pyproject.toml` files updated to allow for Python version `>=3.8`.
diff --git a/...lm/extensions-helm-spark-infrastructure/aissemble-spark-history-chart/README.md b/...lm/extensions-helm-spark-infrastructure/aissemble-spark-history-chart/README.md
@@ -19,6 +19,7 @@ _**NOTE:**_ *the version should match the aiSSEMBLE project version.*
 
 | Property                         | Description                                                                                                                       | Default                                                                                                                  |
 |----------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------|
+| app.name                         | Sets label for app.kubernetes.io/name                                                                                             | Chart.Name (aissemble-spark-history-chart)                                                                               | 
 | enable                           | Enable or disable the entirety of the spark-history-server deployment.  When false, equivalent to not installing the chart.       | true                                                                                                                     |
 | deployment.annotations           | Annotations to apply to the Spark History Server Deployment.                                                                      | {}                                                                                                                       |
 | deployment.labels                | Labels to apply to the Spark History Server Deployment.                                                                           | {}                                                                                                                       |

diff --git a/...tensions-helm-spark-infrastructure/aissemble-spark-history-chart/templates/configmap.yaml b/...tensions-helm-spark-infrastructure/aissemble-spark-history-chart/templates/configmap.yaml
@@ -2,7 +2,7 @@
 apiVersion: v1
 kind: ConfigMap
 metadata:
-  name: {{ .Release.Name }}-shs-spark-config
+  name: {{ .Values.app.name }}-spark-config
 data:
   sparkConf:
     {{- toYaml .Values.sparkConf | nindent 8 }}

diff --git a/...ensions-helm-spark-infrastructure/aissemble-spark-history-chart/templates/deployment.yaml b/...ensions-helm-spark-infrastructure/aissemble-spark-history-chart/templates/deployment.yaml
@@ -2,21 +2,23 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  name: {{ .Release.Name }}-shs
+  name: "{{ .Values.app.name | default .Chart.Name }}"
   {{- if .Values.deployment.annotations }}
   annotations:
     {{ toYaml .Values.deployment.annotations | default "" }}
   {{- end }}
   labels:
-     app: {{ .Release.Name }}
-     {{- if .Values.deployment.labels }}
-     {{ toYaml .Values.deployment.labels }}
-     {{- end }}
+    app.kubernetes.io/instance: "{{ .Values.app.name | default .Chart.Name }}"
+    app.kubernetes.io/name: "{{ .Values.app.name | default .Chart.Name }}"
+    {{- if .Values.deployment.labels }}
+    {{ toYaml .Values.deployment.labels }}
+    {{- end }}
 spec:
   replicas: {{ .Values.deployment.replicas }}
   selector:
     matchLabels:
-      app: {{ .Release.Name }}
+      app.kubernetes.io/instance: "{{ .Values.app.name | default .Chart.Name }}"
+      app.kubernetes.io/name: "{{ .Values.app.name | default .Chart.Name }}"
       {{- if .Values.deployment.labels }}
       {{ toYaml .Values.deployment.labels }}
       {{- end }}
@@ -27,7 +29,8 @@ spec:
         {{ toYaml .Values.deployment.annotations | default "" }}
       {{- end }}
       labels:
-        app: {{ .Release.Name }}
+        app.kubernetes.io/instance: "{{ .Values.app.name | default .Chart.Name }}"
+        app.kubernetes.io/name: "{{ .Values.app.name | default .Chart.Name }}"
         {{- if .Values.deployment.labels }}
         {{ toYaml .Values.deployment.labels }}
         {{- end }}
@@ -116,7 +119,7 @@ spec:
         {{- if not (eq .Values.sparkConf "") }}
         - name: spark-config
           configMap:
-            name: {{ .Release.Name }}-shs-spark-config
+            name: {{ .Values.app.name }}-spark-config
             items:
               - key: sparkConf
                 path: spark-defaults.conf

diff --git a/...extensions-helm-spark-infrastructure/aissemble-spark-history-chart/templates/ingress.yaml b/...extensions-helm-spark-infrastructure/aissemble-spark-history-chart/templates/ingress.yaml
@@ -2,15 +2,14 @@
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
-  name: {{ .Release.Name }}-shs-ingress
+  name: {{ .Values.app.name | default .Chart.Name }}
   {{- with .Values.ingress.metadata.annotations }}
   annotations:
   {{- toYaml . | nindent 8 }}
   {{- end }}
 spec:
   ingressClassName: {{ .Values.ingress.ingressClassName }}
   rules:
-    {{- $releasename := printf "%s-service" .Release.Name }}
     {{- range .Values.ingress.hosts }}
     - host: {{ .host | quote }}
       http:
@@ -20,7 +19,7 @@ spec:
             pathType: {{ .pathType }}
             backend:
               service:
-                name: {{ .backend.service.name | default $releasename }}
+                name: {{ required "A valid .backend.service.name entry required." .backend.service.name }}
                 port:
                   number: {{ .backend.service.port.number }}
           {{- end }}

diff --git a/...extensions-helm-spark-infrastructure/aissemble-spark-history-chart/templates/service.yaml b/...extensions-helm-spark-infrastructure/aissemble-spark-history-chart/templates/service.yaml
@@ -2,14 +2,18 @@
 apiVersion: v1
 kind: Service
 metadata:
-  name: {{ .Release.Name }}-shs-service
+  labels:
+    app.kubernetes.io/instance: {{ .Values.app.name | default .Chart.Name }}
+    app.kubernetes.io/name: {{ .Values.app.name | default .Chart.Name }}
+  name: {{ .Values.app.name | default .Chart.Name }}
   {{- if .Values.service.annotations }}
   annotations: {{ .Values.service.annotations }}
   {{- end }}
 spec:
   type: {{.Values.service.type }}
   selector:
-    app: {{ .Release.Name }}
+    app.kubernetes.io/instance: {{ .Values.app.name | default .Chart.Name }}
+    app.kubernetes.io/name: {{ .Values.app.name | default .Chart.Name }}
   ports:
     - name: {{ .Values.service.port.name }}
       protocol: TCP

diff --git a/...nsions-helm-spark-infrastructure/aissemble-spark-history-chart/tests/deployment_test.yaml b/...nsions-helm-spark-infrastructure/aissemble-spark-history-chart/tests/deployment_test.yaml
@@ -38,14 +38,23 @@ tests:
       deployment.labels.label1: label2
     asserts:
       - equal:
-          path: metadata.labels.app
-          value: RELEASE-NAME
+          path: spec.selector.matchLabels["app.kubernetes.io/instance"]
+          value: aissemble-spark-history-chart
       - equal:
-          path: spec.template.metadata.labels.app
-          value: RELEASE-NAME
+          path: spec.selector.matchLabels["app.kubernetes.io/name"]
+          value: aissemble-spark-history-chart
       - equal:
-          path: spec.selector.matchLabels.app
-          value: RELEASE-NAME
+          path: metadata.labels["app.kubernetes.io/instance"]
+          value: aissemble-spark-history-chart
+      - equal:
+          path: metadata.labels["app.kubernetes.io/name"]
+          value: aissemble-spark-history-chart
+      - equal:
+          path: spec.template.metadata.labels["app.kubernetes.io/instance"]
+          value: aissemble-spark-history-chart
+      - equal:
+          path: spec.template.metadata.labels["app.kubernetes.io/name"]
+          value: aissemble-spark-history-chart
 
   - it: Should apply requested annotations
     set:

diff --git a/...xtensions-helm-spark-infrastructure/aissemble-spark-history-chart/tests/ingress_test.yaml b/...xtensions-helm-spark-infrastructure/aissemble-spark-history-chart/tests/ingress_test.yaml
@@ -39,7 +39,7 @@ tests:
     asserts:
       - equal:
           path: metadata.name
-          value: RELEASE-NAME-shs-ingress
+          value: aissemble-spark-history-chart
       - notExists:
           path: metadata.annotations
 
@@ -78,7 +78,7 @@ tests:
           value: Prefix
       - equal:
           path: spec.rules[0].http.paths[0].backend.service.name
-          value: RELEASE-NAME-service
+          value: spark-history
       - equal:
           path: spec.rules[0].http.paths[0].backend.service.port.number
           value: 18080

diff --git a/...xtensions-helm-spark-infrastructure/aissemble-spark-history-chart/tests/service_test.yaml b/...xtensions-helm-spark-infrastructure/aissemble-spark-history-chart/tests/service_test.yaml
@@ -19,15 +19,18 @@ tests:
           count: 1
       - equal:
           path: metadata.name
-          value: RELEASE-NAME-shs-service
+          value: aissemble-spark-history-chart
       - notExists:
           path: metadata.annotations
       - equal:
           path: spec.type
           value: LoadBalancer
       - equal:
-          path: spec.selector.app
-          value: RELEASE-NAME
+          path: spec.selector["app.kubernetes.io/instance"]
+          value: aissemble-spark-history-chart
+      - equal:
+          path: spec.selector["app.kubernetes.io/name"]
+          value: aissemble-spark-history-chart
       - contains:
           path: spec.ports
           content:

diff --git a/...nsions-helm-spark-infrastructure/aissemble-spark-history-chart/tests/spark_conf_test.yaml b/...nsions-helm-spark-infrastructure/aissemble-spark-history-chart/tests/spark_conf_test.yaml
@@ -15,6 +15,7 @@ tests:
   - it: If spark conf is specified, the ConfigMap should be created with the provided values
     set:
       sparkConf: "spark.executor.instances: 2"
+      app.name: "spark-history"
     templates:
       - configmap.yaml
     asserts:
@@ -25,14 +26,15 @@ tests:
       - notFailedTemplate: {}
       - equal:
           path: metadata.name
-          value: "RELEASE-NAME-shs-spark-config"
+          value: "spark-history-spark-config"
       - equal:
           path: data.sparkConf
           value: "spark.executor.instances: 2"
 
   - it: If spark conf is specified, a volume should be created from the created configmap
     set:
       sparkConf: "spark.executor.instances: 2"
+      app.name: "spark-history"
     templates:
       - deployment.yaml
     asserts:
@@ -41,7 +43,7 @@ tests:
           content:
             name: spark-config
             configMap:
-              name: RELEASE-NAME-shs-spark-config
+              name: spark-history-spark-config
               items:
                 - key: sparkConf
                   path: spark-defaults.conf

diff --git a/...sions-helm/extensions-helm-spark-infrastructure/aissemble-spark-history-chart/values.yaml b/...sions-helm/extensions-helm-spark-infrastructure/aissemble-spark-history-chart/values.yaml
@@ -1,4 +1,7 @@
 # This file contains the default values for the aiSSEMBLE Spark History Helm chart.
+app:
+  # Sets labels for app.kubernetes.io/name Default is Chart.Name (aissemble-spark-history-chart)
+  name: ''
 
 enable: true
 
@@ -42,6 +45,7 @@ ingress:
           pathType: Prefix
           backend:
             service:
+              name: spark-history
               port:
                 number: 18080
 

diff --git a/...lm/extensions-helm-spark-infrastructure/aissemble-thrift-server-chart/README.md b/...lm/extensions-helm-spark-infrastructure/aissemble-thrift-server-chart/README.md
@@ -19,6 +19,7 @@ _**NOTE:**_ *the version should match the aiSSEMBLE project version.*
 
 | Property                         | Description                                                                                                                       | Default                                                               |
 |----------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------|
+| app.name                         | Sets label for app.kubernetes.io/name                                                                                             | Chart.Name (aissemble-thrift-server-chart)                            |
 | enable                           | Enable or disable the entirety of the spark-thrift-server deployment.  When false, equivalent to not installing the chart.        | true                                                                  |
 | deployment.annotations           | Annotations to apply to the Spark Thrift Server Deployment.                                                                       | {}                                                                    |
 | deployment.labels                | Labels to apply to the Spark Thrift Server Deployment.                                                                            | {}                                                                    |

diff --git a/...ensions-helm-spark-infrastructure/aissemble-thrift-server-chart/templates/deployment.yaml b/...ensions-helm-spark-infrastructure/aissemble-thrift-server-chart/templates/deployment.yaml
@@ -2,21 +2,23 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  name: {{ .Release.Name }}-sts
+  name: "{{ .Values.app.name | default .Chart.Name }}"
   {{- if .Values.deployment.annotations }}
   annotations:
     {{ toYaml .Values.deployment.annotations | default "" }}
   {{- end }}
   labels:
-     app: {{ .Release.Name }}
-     {{- if .Values.deployment.labels }}
-     {{ toYaml .Values.deployment.labels }}
-     {{- end }}
+    app.kubernetes.io/instance: "{{ .Values.app.name | default .Chart.Name }}"
+    app.kubernetes.io/name: "{{ .Values.app.name | default .Chart.Name }}"
+    {{- if .Values.deployment.labels }}
+    {{ toYaml .Values.deployment.labels }}
+    {{- end }}
 spec:
   replicas: {{ .Values.deployment.replicas }}
   selector:
     matchLabels:
-      app: {{ .Release.Name }}
+      app.kubernetes.io/instance: "{{ .Values.app.name | default .Chart.Name }}"
+      app.kubernetes.io/name: "{{ .Values.app.name | default .Chart.Name }}"
       {{- if .Values.deployment.labels }}
       {{ toYaml .Values.deployment.labels }}
       {{- end }}
@@ -27,14 +29,15 @@ spec:
         {{ toYaml .Values.deployment.annotations | default "" }}
       {{- end }}
       labels:
-        app: {{ .Release.Name }}
+        app.kubernetes.io/instance: "{{ .Values.app.name | default .Chart.Name }}"
+        app.kubernetes.io/name: "{{ .Values.app.name | default .Chart.Name }}"
         {{- if .Values.deployment.labels }}
         {{ toYaml .Values.deployment.labels }}
         {{- end }}
     spec:
       {{- if or (not (empty .Values.dependencies.packages)) (not (empty .Values.dependencies.jars)) }}
       initContainers:
-        - name: "populate-sts-jar-volume"
+        - name: "populate-thrift-service-jar-volume"
           image: {{ .Values.deployment.image.repository }}:{{ .Values.deployment.image.tag }}
           imagePullPolicy: {{ .Values.deployment.image.imagePullPolicy | default "IfNotPresent" }}
           command: ["sh", "-c", "cp /opt/spark/jars/* /tmp/jars/"]

diff --git a/...extensions-helm-spark-infrastructure/aissemble-thrift-server-chart/templates/ingress.yaml b/...extensions-helm-spark-infrastructure/aissemble-thrift-server-chart/templates/ingress.yaml
@@ -2,15 +2,14 @@
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
-  name: {{ .Release.Name }}-sts-ingress
+  name: {{ .Values.app.name | default .Chart.Name }}
   {{- with .Values.ingress.metadata.annotations }}
   annotations:
   {{- toYaml . | nindent 8 }}
   {{- end }}
 spec:
   ingressClassName: {{ .Values.ingress.ingressClassName }}
   rules:
-    {{- $releasename := printf "%s-service" .Release.Name }}
     {{- range .Values.ingress.hosts }}
     - host: {{ .host | quote }}
       http:
@@ -20,7 +19,7 @@ spec:
             pathType: {{ .pathType }}
             backend:
               service:
-                name: {{ .backend.service.name | default $releasename }}
+                name: {{ .backend.service.name }}
                 port:
                   number: {{ .backend.service.port.number }}
           {{- end }}

diff --git a/...extensions-helm-spark-infrastructure/aissemble-thrift-server-chart/templates/service.yaml b/...extensions-helm-spark-infrastructure/aissemble-thrift-server-chart/templates/service.yaml
@@ -2,14 +2,18 @@
 apiVersion: v1
 kind: Service
 metadata:
-  name: {{ .Release.Name }}-sts-service
+  labels:
+    app.kubernetes.io/instance: {{ .Values.app.name | default .Chart.Name }}
+    app.kubernetes.io/name: {{ .Values.app.name | default .Chart.Name }}
+  name: {{ .Values.app.name | default .Chart.Name }}
   {{- if .Values.service.annotations }}
   annotations: {{ .Values.service.annotations }}
   {{- end }}
 spec:
   type: {{.Values.service.type }}
   selector:
-    app: {{ .Release.Name }}
+    app.kubernetes.io/instance: {{ .Values.app.name | default .Chart.Name }}
+    app.kubernetes.io/name: {{ .Values.app.name | default .Chart.Name }}
   ports:
     {{ .Values.service.ports | toYaml | nindent 4}}
 {{- end }}