Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix restart policy bug in mpi job UpdateJobConditions #2344

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

fyxemmmm
Copy link

@fyxemmmm fyxemmmm commented Dec 5, 2024

What this PR does / why we need it:
This PR fixes an issue in the MPIJob controller logic related to the handling of failed replicas. Specifically, the condition for determining the restart policy was incorrect.

Previously, the code used:
if spec.RestartPolicy == kubeflowv1.RestartPolicyExitCode
This condition was problematic because it failed to handle cases where the restart policy should be based on other valid configurations. The incorrect logic could lead to unexpected behavior, such as jobs failing instead of restarting.

The updated code uses:
if spec.RestartPolicy != kubeflowv1.RestartPolicyNever
This ensures that jobs are restarted appropriately unless explicitly configured not to restart, aligning the behavior with the intended design and user expectations.

Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign johnugeorge for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@fyxemmmm fyxemmmm mentioned this pull request Dec 5, 2024
@coveralls
Copy link

Pull Request Test Coverage Report for Build 12176368662

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall first build on fangyuxuan/fix_mpi_bug at 100.0%

Totals Coverage Status
Change from base Build 12166368274: 100.0%
Covered Lines: 77
Relevant Lines: 77

💛 - Coveralls

Copy link
Member

@tenzen-y tenzen-y left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was not sure if this is really bugs.
Could you add tests to prove the bug situations?
https://github.com/kubeflow/training-operator/blob/master/pkg/controller.v1/mpi/mpijob_controller_test.go

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants