fix MpiJob status inaccuracy when kill redundant worker replicas

[PeterChg]

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #

Checklist:

• Docs included if any changes are user facing

[PeterChg]

/assign @Jeffwan

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: PeterChg
To complete the pull request process, please ask for approval from jeffwan after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coveralls]

Pull Request Test Coverage Report for Build 2546467435

• 4 of 4 (100.0%) changed or added relevant lines in 1 file are covered.
• 2 unchanged lines in 1 file lost coverage.
• Overall coverage increased (+0.03%) to 39.979%

---

Files with Coverage Reduction	New Missed Lines	%
pkg/controller.v1/mpi/mpijob_controller.go	2	77.7%

Totals
Change from base Build 2531249966:	0.03%
Covered Lines:	2334
Relevant Lines:	5838

---

💛 - Coveralls

[terrytangyuan]

Could you elaborate why there will be a redundant worker replica?

[PeterChg]

Could you elaborate why there will be a redundant worker replica?

/cc @gaocegege
/cc @zw0610

I see that this scenario is considered in the code, i also want to know why this will happen

[zw0610]

If you are referring Elastic Training with MPIJob, I believe the case could be:

With elastic training enabled, when some worker Pods is failed, evicted or deleted, the MPIJob should continue to run.

But I'm not sure whether we can describe such case as redundant workers.

@PeterChg When you say this pr is to fix something, as in the title, could you be more specific about which issue this pull request is going to fix? You can just adding the description in this pull request, link an existing issue or create a new issue.

[PeterChg]

If you are referring Elastic Training with MPIJob, I believe the case could be:

With elastic training enabled, when some worker Pods is failed, evicted or deleted, the MPIJob should continue to run.

But I'm not sure whether we can describe such case as redundant workers.

@PeterChg When you say this pr is to fix something, as in the title, could you be more specific about which issue this pull request is going to fix? You can just adding the description in this pull request, link an existing issue or create a new issue.

/cc @terrytangyuan
@zw0610 said is more accurate that what this pr is to fix. I have modified the pr comment.
without this pr ,when Pods is evicted or deleted , the phase of pods will become Failed Before disappears, This will result in a failed MpiJob state.

[johnugeorge]

@zw0610 Do you want to take this in the current release?
/cc @terrytangyuan

Related: #1622

[zw0610]

@johnugeorge I'm afraid we have to wait for the next release cycle because:

1. the description of this pull request is still missing, which could be confusing for future developers
2. I cannot recall situations of some worker reached Succeed status instead of failed, which means I failed to understand the change of this pull request.

[johnugeorge]

Sounds good. We will hold it till next release
/hold