Experiments execution fails due to non-initialized securityContext in injected container

[vpavlin]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

• Deploy Katib using Kubeflow on OpenShift 4.x
• Submit https://github.com/kubeflow/katib/blob/master/examples/v1alpha3/random-example.yaml
• In kubectl describe jobs:

  Warning  FailedCreate  2s    job-controller  Error creating: pods "random-example-vvtcwr95-fvf9k" is forbidden: unable to validate against any security context constraint: []

The core issue is that the mutating.pod.katib.kubeflow.org injects a container after some other webhook had populated securityContext in container spec. This "bare" container (https://github.com/kubeflow/katib/blob/master/pkg/webhook/v1alpha3/pod/inject_webhook.go#L181) does not have securityContext initialized and OpenShift will prevent it from start unless the serviceAccount is privileged (which is, on the other hand, a terrible practice:) ).

The ideal solution would be to use reinvocationPolicy (https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#reinvocation-policy), but that is not available in OpenShift 4.2 and the webhook builder used by Katib Controller does not support it either.

My proposed solution then is to use the securityContext from the container that is already present in the pod.

What did you expect to happen:

Pods for Trial jobs are successfully started

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

OpenShift 4.x