#251 remove custom spark operator compilation #253
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ewilkins-csi
commented
Aug 1, 2024
| @@ -31,7 +31,6 @@ | |||
| <version.assembly.plugin>3.6.0</version.assembly.plugin> | |||
| <version.buildhelper.plugin>3.2.0</version.buildhelper.plugin> | |||
| <version.buildnumber.plugin>3.1.0</version.buildnumber.plugin> | |||
| <version.cucumber.reporting.plugin>5.7.5</version.cucumber.reporting.plugin> | |||
There was a problem hiding this comment.
I: Just consolidating the two properties into the common parent pom.
ewilkins-csi
commented
Aug 1, 2024
| COPY --from=builder --chmod=755 /usr/bin/spark-operator /usr/bin/ | ||
| COPY --from=builder --chmod=755 /usr/bin/entrypoint.sh /usr/bin/ | ||
| RUN apt-get update \ | ||
| && apt-get install -y tini \ |
There was a problem hiding this comment.
I: Spark Operator 1.6+ no longer uses the gencerts script or the webhook init/cleanup k8s jobs. So we only need tini as it's used in entrypoint.sh to launch the service.
ewilkins-csi
commented
Aug 1, 2024
| @@ -10,7 +10,7 @@ sources: | |||
| - https://github.com/boozallen/aissemble | |||
| dependencies: | |||
| - name: spark-operator | |||
| version: 1.1.27 | |||
| version: 1.4.6 | |||
There was a problem hiding this comment.
I: Latest version as of today
ewilkins-csi
commented
Aug 1, 2024
ewilkins-csi
commented
Aug 1, 2024
| @@ -48,7 +43,7 @@ spark-operator: | |||
| - name: spark-logging | |||
| mountPath: "/tmp/spark-events" | |||
| - name: ivy-cache | |||
| mountPath: "/opt/spark/.ivy2" | |||
| mountPath: "/home/spark/.ivy2" | |||
There was a problem hiding this comment.
I: The base Spark Operator image uses the default home location for the spark user. Technically we could have updated the home in the image build with RUN usermod, but this sets us up to remove the aissemble-spark-operator image entirely after the upgrade to Spark 3.5.
ewilkins-csi
force-pushed
the
251-remove-custom-spark-operator
branch
from
9bfcabe to
fd4a830
Compare
extensions/extensions-helm/aissemble-spark-operator-chart/values.template.yaml
Show resolved
Hide resolved
carter-cundiff
approved these changes
Aug 1, 2024
- remove the custom compilation of Spark Operator - update the Spark Operator chart dependency to the latest version - modify the Spark Operator chart to work with the stock kubeflow image - remove custom RBAC - change ivy cache mount location Note that the only thing preventing the complete removal of our custom Spark Operator image is the fact that kubeflow's Spark Operator is on Spark 3.5 while we're on Spark 3.4. The upgrade is actually pretty easy without many breaking changes at all, however it started to require some dependency untangling with different versions being pulled in between Spark 3.5 and Quarkus 2.8. Since we know we'll be upgrading Quarkus soon, I'm leaving the Spark 3.5 upgrade to follow that. This should cut a significant amount of time off of the image build though, as the custom Go compilation was taking at least half of the build time.
ewilkins-csi
force-pushed
the
251-remove-custom-spark-operator
branch
from
fd4a830 to
8ff8bc9
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note that the only thing preventing the complete removal of our custom Spark Operator image is the fact that kubeflow's Spark Operator is on Spark 3.5 while we're on Spark 3.4. The upgrade is actually pretty easy without many breaking changes at all, however it started to require some dependency untangling with different versions being pulled in between Spark 3.5 and Quarkus 2.8. Since we know we'll be upgrading Quarkus soon, I'm leaving the Spark 3.5 upgrade to follow that. This should cut a significant amount of time off of the image build though, as the custom Go compilation was taking at least half of the build time.