Adding an option to set the priority class for spark-operator pod

[pkgajulapalli]

Purpose of this PR

Adding an option to set the priority class for spark-operator pod. This will help in better managing the preemption of pods.

Proposed changes:

• Added an optional value in helm chart priorityClassName for spark-operator pod.

Change Category

Indicate the type of change by marking the applicable boxes:

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Checklist

Before submitting your PR, please review the following:

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

1. Created a custom priority class.

$ k8s get priorityclass custom-priority-class -o yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: custom-priority-class
preemptionPolicy: PreemptLowerPriority
value: 10000

2. Installed helm chart with values specifying the same priority class

image:
  repository: "my-spark-operator"
  tag: "latest"
priorityClassName: "custom-priority-class"

3. spark-operator pod started with custom priority.

$ k8s describe pod spark-operator-6cfbb57c-bn5n6 | grep -i 'priority'
Priority:             10000
Priority Class Name:  custom-priority-class

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[vara-bonthu]

Thanks for the PR!

Could you please uplift the Chart version with this change?
Provide any test results if you have any.

[vara-bonthu]

/lgtm

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vara-bonthu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vara-bonthu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment