How to restart the training JOB when one training process fails in cluster environment to recover the training?

[kevinsummer219]

What you would like to be added?

I use pytorchJOB.

Why is this needed?

I think, that is valid idea. So in such restart policy our controller should re-create all PyTorchJob's pods in case of single Pod failure.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[kevinsummer219]

The current situation is that when a Pod encounters an exception, it can be automatically restarted. However, as for multi-node training, all Pods need to be restarted. How do I configure this YAML file? Thanks!

[kevinsummer219]

The pytorchJOB YAML configuration file I used：
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
name: train-poc
namespace: default
labels:
kueue.x-k8s.io/queue-name: user-queue
spec:
runPolicy:
backoffLimit: 5
nprocPerNode: gpu
pytorchReplicaSpecs:
Worker:
replicas: 2
restartPolicy: OnFailure
template:
metadata:

[kannon92]

I believe that the v2 Training Operator will solve this by utilizing JobSet Failure Policies.

I'm unsure of the path for v1 (@tenzen-y @andreyvelich?).

[andreyvelich]

@kubeflow/wg-training-leads @kuizhiqing can we leverage the RestartPolicy API in V1 to restart all replica's pods in case of failure ?

Yes, with V2, we can use FailurePolicy within JobSet.