Cherry pick of #2350 #2355 #2357 #2344 #2358 #2360 into release-0.17 branch #667

Job	Run time
e2e (v1.27.11, long-running-resume,from-volume-resume,median-stop)	13m 52s
e2e (v1.27.11, grid,bayesian-optimization,tpe,multivariate-tpe,cma-es,hyperband)	24m 11s
e2e (v1.27.11, file-metrics-collector,pytorchjob-mnist)	13m 38s
e2e (v1.27.11, median-stop-with-json-format,file-metrics-collector-with-json-format)	15m 5s
e2e (v1.28.7, long-running-resume,from-volume-resume,median-stop)	17m 33s
e2e (v1.28.7, grid,bayesian-optimization,tpe,multivariate-tpe,cma-es,hyperband)	21m 14s
e2e (v1.28.7, file-metrics-collector,pytorchjob-mnist)	12m 7s
e2e (v1.28.7, median-stop-with-json-format,file-metrics-collector-with-json-format)	11m 29s
e2e (v1.29.2, long-running-resume,from-volume-resume,median-stop)	17m 2s
e2e (v1.29.2, grid,bayesian-optimization,tpe,multivariate-tpe,cma-es,hyperband)	22m 39s
e2e (v1.29.2, file-metrics-collector,pytorchjob-mnist)	12m 32s
e2e (v1.29.2, median-stop-with-json-format,file-metrics-collector-with-json-format)	13m 8s
	3h 14m 30s

Provide feedback