Fixes a bug with dynamic allocation forcing the executor count to be 1 even when minExecutors is set to 0

[peter-mcclonski]

🛑 Important:

Please open an issue to discuss significant work before you start. We appreciate your contributions and don't want your efforts to go to waste!

For guidelines on how to contribute, please review the CONTRIBUTING.md document.

Purpose of this PR

Fixes a bug with dynamic allocation forcing the executor count to be 1 even when minExecutors is set to 0

Change Category

Indicate the type of change by marking the applicable boxes:

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

Resolves otherwise unexpected behavior where the value of minExecutors was not fully respected.

Checklist

Before submitting your PR, please review the following:

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

[peter-mcclonski]

Resolves #1779

[peter-mcclonski]

@vara-bonthu some quick feedback:

• The build-api-docs pre-commit check isn't possible to resolve in this case. It's getting flagged because there is a code difference, but the difference isn't one that would be reflectable in the api docs. Regenerating the api-docs results in no change.
• The Chart appVersion has been updated, but the build-helm-chart and build-spark-operator checks don't seem to know how to handle it.

Honestly not sure how to resolve these issues, since it looks like the actual resolution isn't being respected by the check. Can you advise?

[vara-bonthu]

Did you try to rebase your PR branch with master and uplift the chat version?

[peter-mcclonski]

Did you try to rebase your PR branch with master and uplift the chat version?

Yup, I had tried this previously but just re-attempted for the next time the workflow is run. I don't think that will/would assist with the api-docs issue.

[vara-bonthu]

/approve

@yuchaoran2011 please review the PR

[peter-mcclonski]

@vara-bonthu Any hope of getting this merged soon?

[vara-bonthu]

Waiting for another approval from @yuchaoran2011

[vikas-saxena02]

/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vara-bonthu, vikas-saxena02

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vara-bonthu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[yuchaoran2011]

/lgtm