Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: support driver and executor pod use different priority #2146

Merged
merged 15 commits into from
Sep 10, 2024
Merged

feat: support driver and executor pod use different priority #2146

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
32aa2b4
feat: support driver and executor pod use different priority
Kevinmmt Aug 27, 2024
bd7c2eb
feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.Pr…
Kevinmmt Aug 29, 2024
392e3e8
feat: merge the logic of setPodPriorityClassName into addPriorityClas…
Kevinmmt Aug 29, 2024
a791e07
feat: support driver and executor pod use different priority
Kevinmmt Aug 27, 2024
4ccb6b9
feat: if *app.Spec.Driver.PriorityClassName and *app.Spec.Executor.Pr…
Kevinmmt Aug 29, 2024
2a9b69c
feat: merge the logic of setPodPriorityClassName into addPriorityClas…
Kevinmmt Aug 29, 2024
25fb23f
feat: add adjust pointer if is nil
Kevinmmt Aug 29, 2024
a39ae87
feat: remove spec.batchSchedulerOptions.priorityClassName define , sp…
Kevinmmt Sep 9, 2024
b781c3c
feat: remove spec.batchSchedulerOptions.priorityClassName define , sp…
Kevinmmt Sep 9, 2024
921d31f
feat: remove spec.batchSchedulerOptions.priorityClassName define , sp…
Kevinmmt Sep 9, 2024
74dbae6
feat: Optimize code to avoid null pointer exceptions
Kevinmmt Sep 9, 2024
a8d27ee
fix: remove backup crd files
Kevinmmt Sep 9, 2024
d4278ce
Merge remote-tracking branch 'upstream/master'
Kevinmmt Sep 9, 2024
ef6b393
fix: remove BatchSchedulerOptions.PriorityClassName test code
Kevinmmt Sep 10, 2024
84ed21c
fix: add driver and executor pod priorityClassName test code
Kevinmmt Sep 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions api/v1beta2/sparkapplication_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -536,6 +536,9 @@ type DriverSpec struct {
// Ports settings for the pods, following the Kubernetes specifications.
// +optional
Ports []Port `json:"ports,omitempty"`
// PriorityClassName is the name of the PriorityClass for the driver pod.
// +optional
PriorityClassName *string `json:"priorityClassName,omitempty"`
}

// ExecutorSpec is specification of the executor.
Expand Down Expand Up @@ -563,6 +566,9 @@ type ExecutorSpec struct {
// Ports settings for the pods, following the Kubernetes specifications.
// +optional
Ports []Port `json:"ports,omitempty"`
// PriorityClassName is the name of the PriorityClass for the executor pod.
// +optional
PriorityClassName *string `json:"priorityClassName,omitempty"`
}

// NamePath is a pair of a name and a path to which the named objects should be mounted to.
Expand Down
10 changes: 10 additions & 0 deletions api/v1beta2/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions charts/spark-operator-chart/crds/sparkoperator.k8s.io_scheduledsparkapplications.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3179,6 +3179,10 @@ spec:
- protocol
type: object
type: array
priorityClassName:
description: PriorityClassName is the name of the PriorityClass
for the driver pod.
type: string
schedulerName:
description: SchedulerName specifies the scheduler that will
be used for scheduling
Expand Down Expand Up @@ -7946,6 +7950,10 @@ spec:
- protocol
type: object
type: array
priorityClassName:
description: PriorityClassName is the name of the PriorityClass
for the executor pod.
type: string
schedulerName:
description: SchedulerName specifies the scheduler that will
be used for scheduling
Expand Down
8 changes: 8 additions & 0 deletions charts/spark-operator-chart/crds/sparkoperator.k8s.io_sparkapplications.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3127,6 +3127,10 @@ spec:
- protocol
type: object
type: array
priorityClassName:
description: PriorityClassName is the name of the PriorityClass
for the driver pod.
type: string
schedulerName:
description: SchedulerName specifies the scheduler that will be
used for scheduling
Expand Down Expand Up @@ -7864,6 +7868,10 @@ spec:
- protocol
type: object
type: array
priorityClassName:
description: PriorityClassName is the name of the PriorityClass
for the executor pod.
type: string
schedulerName:
description: SchedulerName specifies the scheduler that will be
used for scheduling
Expand Down
8 changes: 8 additions & 0 deletions config/crd/bases/sparkoperator.k8s.io_scheduledsparkapplications.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3179,6 +3179,10 @@ spec:
- protocol
type: object
type: array
priorityClassName:
description: PriorityClassName is the name of the PriorityClass
for the driver pod.
type: string
schedulerName:
description: SchedulerName specifies the scheduler that will
be used for scheduling
Expand Down Expand Up @@ -7946,6 +7950,10 @@ spec:
- protocol
type: object
type: array
priorityClassName:
description: PriorityClassName is the name of the PriorityClass
for the executor pod.
type: string
schedulerName:
description: SchedulerName specifies the scheduler that will
be used for scheduling
Expand Down
8 changes: 8 additions & 0 deletions config/crd/bases/sparkoperator.k8s.io_sparkapplications.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3127,6 +3127,10 @@ spec:
- protocol
type: object
type: array
priorityClassName:
description: PriorityClassName is the name of the PriorityClass
for the driver pod.
type: string
schedulerName:
description: SchedulerName specifies the scheduler that will be
used for scheduling
Expand Down Expand Up @@ -7864,6 +7868,10 @@ spec:
- protocol
type: object
type: array
priorityClassName:
description: PriorityClassName is the name of the PriorityClass
for the executor pod.
type: string
schedulerName:
description: SchedulerName specifies the scheduler that will be
used for scheduling
Expand Down
26 changes: 16 additions & 10 deletions internal/webhook/sparkpod_defaulter.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -475,21 +475,27 @@ func addSchedulerName(pod *corev1.Pod, app *v1beta2.SparkApplication) error {
return nil
}

func setPodPriorityClassName(pod *corev1.Pod, priorityClassName *string) {
if priorityClassName != nil && *priorityClassName != "" {
pod.Spec.PriorityClassName = *priorityClassName
pod.Spec.Priority = nil
pod.Spec.PreemptionPolicy = nil
}
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we could merge the logic of setPodPriorityClassName into addPriorityClassName.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ChenYi015 Already adapted merge the logic of setPodPriorityClassName into addPriorityClassName, thanks for ur suggestions

func addPriorityClassName(pod *corev1.Pod, app *v1beta2.SparkApplication) error {
var priorityClassName *string
if app.Spec.BatchSchedulerOptions != nil {

if util.IsDriverPod(pod) && *app.Spec.Driver.PriorityClassName != "" {
priorityClassName = app.Spec.Driver.PriorityClassName
} else if util.IsExecutorPod(pod) && *app.Spec.Executor.PriorityClassName != "" {
priorityClassName = app.Spec.Executor.PriorityClassName
} else if app.Spec.BatchSchedulerOptions != nil {
priorityClassName = app.Spec.BatchSchedulerOptions.PriorityClassName
}

if priorityClassName != nil && *priorityClassName != "" {
pod.Spec.PriorityClassName = *priorityClassName
if pod.Spec.Priority != nil {
pod.Spec.Priority = nil
}
if pod.Spec.PreemptionPolicy != nil {
pod.Spec.PreemptionPolicy = nil
}
}
setPodPriorityClassName(pod, priorityClassName)

return nil
}

Expand Down