Pod stuck in Pending due to kubelet skip to create pod sandbox

[yqwang-ms]

Organization Name:

Short summary about the issue/question:
kubernetes/kubernetes#79398
kubernetes/kubernetes#79451

Long term solution is (1.17 k8s) kubernetes/kubernetes@3fac48f

Short term workaround:
Change Pod restartPolicy=OnFailure for elastic job (TBD: and training job)

Brief what process you are following:

How to reproduce it:

OpenPAI Environment:

• OpenPAI version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Hardware (e.g. core number, memory size, storage size, GPU type etc.):
• Others:

Anything else we need to know:

related log:

kuberuntime_manager.go:427] No ready sandbox for pod "..." can be found. Need to start a new one

[yqwang-ms]

@abuccts pls help to do the change for elastic job :)

[yqwang-ms]

let's hold on this first to test if 1.16.0 can be in this release

[yqwang-ms]

Tracked in #3863.