TASK: Overwrite the ivy user directory in the spark-application chart

[csun-cpointe]

Description

For java 17 upgrade, we have upgraded the aissemble-spark base image, which has a different home directory than the /opt/spark/ directory so to make sure Ivy cache can find the right directory (Ivy cache is default to the environment's home directory. e.g.: ~/.ivy2), we will need to override the ivy user directory by specifying the spark.jars.ivy directory in the chart.

Currently, to fix the build issue, we have moved the spark.jars.ivy from the spark application chart to the dev values file to avoid the local test issue. However, since our local test is not using the docker container but only copy over the spark configuration from the yaml file to configure the spark session in the local environment, we should not copy spark.jars.ivy directory for the spark tester setup but leave it as default so that it can still use the default home directory and when we deploying the spark application to different environment, we don't need to set spark.jars.ivy path in the values file.

DOD

Acceptance criteria required to complete the work

• Move the spark.jars.ivy configuration to the aissemble-spark-application chart
• In python or java test code, avoid spark.jars.ivy configuration being setup in the spark

Test Strategy/Script

How will this item be verified?

• Pull the latest aissemble repo

• OTS Only:

  • Since the aissemble-spark-application-chart is not being pushed up to GitHub packages yet, we will make some local modification
    • In thefoundation/foundation-mda/src/main/resources/templates/data-delivery-common/pipeline.chart.pom.configuration.vm, modify the line 32 to point to your local directory <<path to your baseline repo>>/extensions/extensions-helm/aissemble-spark-application-chart
    • In the foundation/foundation-mda/src/main/resources/templates/data-delivery-common/pipeline.chart.pom.configuration.vm, modify the line 24 to be {path = "/path to your baseline repo/extensions/extensions-data-delivery/aissemble-extensions-data-delivery-spark-py", develop = true}
    • Run mvn clean install -pl :foundation-mda -Dmaven.build.cache.skipCache

• Run mvn -B clean install -Parchetype-test -pl :foundation-archetype and verify the build is success.

References/Additional Context

As needed

[ewilkins-csi]

DoD/OTS looks good ✅

[carter-cundiff]

Testing passed: