privileged operation doesn't work

[donbowman]

If I create a privileged pod running dind (as eg we might do if using gitlab-runner):

kubectl apply -f - << EOF
apiVersion: v1                                                                                                                                               
kind: Pod
metadata:
  name: dind
  labels:
    role: dind
spec:
  containers:
    - name: dind
      image: docker:dind
      securityContext:
        privileged: true
EOF

And then I kubectl exec into it and run:

# docker pull alpine
# docker run --rm -it alpine sh               

It ends up not working:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"could not create session key: operation not permitted\"": unknown.

In the dind container we see this log:

time="2018-10-27T21:07:10Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/7428063ad54a9fafb5bf377d36f2f2c977472b67ae0001854c6e71c533a22f27/shim.sock" debug=false pid=305 
time="2018-10-27T21:07:10Z" level=info msg="shim reaped" id=7428063ad54a9fafb5bf377d36f2f2c977472b67ae0001854c6e71c533a22f27                                 
time="2018-10-27T21:07:10.253654428Z" level=error msg="stream copy error: reading from a closed fifo"                                                        
time="2018-10-27T21:07:10.498102350Z" level=error msg="7428063ad54a9fafb5bf377d36f2f2c977472b67ae0001854c6e71c533a22f27 cleanup: failed to delete container from containerd: no such container"
time="2018-10-27T21:07:10.512689130Z" level=error msg="Handler for POST /v1.38/containers/7428063ad54a9fafb5bf377d36f2f2c977472b67ae0001854c6e71c533a22f27/start returned error: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"could not create session key: operation not permitted\\\"\": unknown"

[donbowman]

This is because: # echo 123 | keyctl padd user foobar @u
fails in the systemd-nspawn'd containers with
add_key: Operation not permitted

https://bugzilla.redhat.com/show_bug.cgi?id=1138601 --> keyctl & add_key are not namespaced

systemd/systemd#5163

Adding

+               "--system-call-filter=@keyring",
+               "--private-users=false",
+               "--bind=/sys/kernel/security

to pkg/cnispawn/spawn.go causes it to get further, now getting:

time="2018-10-27T21:53:49.048927738Z" level=error msg="Handler for POST /v1.38/containers/6602777084c219553f487d3c6aa46e831a59a1e13752020f056b05c13d764c52/start returned error: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"rootfs_linux.go:58: mounting \\\\\\\"cgroup\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/overlay2/8ae264b30d0bd8a1b19d5f02c73e04def0629f584e48e956016dca0dbbdce27a/merged\\\\\\\" at \\\\\\\"/sys/fs/cgroup\\\\\\\" caused \\\\\\\"stat /sys/machine-kube\\\\\\\\\\\\\\\\x2dspawn\\\\\\\\\\\\\\\\x2dflannel\\\\\\\\\\\\\\\\x2dworker\\\\\\\\\\\\\\\\x2d691g4a.scope/system.slice/containerd.service/docker/6602777084c219553f487d3c6aa46e831a59a1e13752020f056b05c13d764c52: no such file or directory\\\\\\\"\\\"\": unknown"

[dongsupark]

Thanks for the report.

I've just followed what you described, and I can confirm that your patch above fixes the issue of keyring.
Can you please create a PR to fix that?

You also mentioned there were another issue with cgroups, even after applying the keyring patch.
That issue could be probably fixed by #328.
With the 2 patches applied, your issue would be probably gone.

[dongsupark]

#328 was merged, so I suppose your issue would be gone.
If there's still the same issue, feel free to reopen it.