SElinux not correctly working with overlayfs

[cyrus-mc]

Environment

OS: CentOS 7.1
Kernel: 3.10.0
rkt: 0.10.0
systemd: 208

When trying to start any container, with selinux set to Enforcing I received the following error:

/usr/lib/systemd/systemd: error while loading shared libraries: libselinux.so.1: cannot open shared object file: Permission denied

Disabling selinux (disabled or Permissive) solved the issue.

[yifan-gu]

cc @mjg59 @steveej @jonboulle

[yifan-gu]

From irc discussion, @cyrus-mc is using stage1-coreos.aci

[jonboulle]

hmm, is this the v0.10.0 release from GitHub, or did you build it yourself?
Could you please paste the entire output of what happens when you try to run it?

[ericchiang]

I've gotten the same issue when using rkt v0.10.0 and v0.11.0 (from releases page) on Fedora 23. I have to disable SELinux to run pods.

Run output:

$ sudo rkt run --insecure-skip-verify sha512-3f25067fc69f
rkt: using image from local store for image name coreos.com/rkt/stage1-coreos:0.11.0
/usr/lib/systemd/systemd: error while loading shared libraries: libselinux.so.1: cannot open shared object file: Permission denied

Full audit logs here.

Interesting part:

type=AVC msg=audit(1447866807.475:2032): avc:  denied  { read } for  pid=11851 comm="systemd" name="libselinux.so.1" dev="dm-2" ino=6569480 scontext=system_u:system_r:svirt_lxc_net_t:s0:c330,c605 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0
type=AVC msg=audit(1447866807.475:2033): avc:  denied  { read } for  pid=11851 comm="systemd" name="libselinux.so.1" dev="dm-2" ino=6569480 scontext=system_u:system_r:svirt_lxc_net_t:s0:c330,c605 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0
type=USER_AVC msg=audit(1447866807.516:2034): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='Unknown permission stop for class system exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
type=USER_AVC msg=audit(1447866807.517:2035): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='Unknown permission stop for class system exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'

[jonboulle]

@mjg59 any ideas here?

[krnowak]

From OOB communication with @mjg59 we can say that rkt is not supported by Fedora/RHEL/CentOS selinux policy.

So, either we add a disable-selinux option to configure or we wait until Fedora policy gets fixed.

Another issue might be that we expect overlays to support selinux and CoreOS has a patched kernel for that.

[alban]

The overlay fs patch for SELinux support is here with others in the series.

[alban]

@lsm5 @rhatdan @mjg59

The command ausearch -m avc -ts recent returns:

----
time->Tue Jan 19 14:54:36 2016
type=AVC msg=audit(1453211676.627:617): avc:  denied  { read } for  pid=7201 comm="systemd" name="libselinux.so.1" dev="dm-1" ino=541346 scontext=system_u:system_r:svirt_lxc_net_t:s0:c9,c955 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0
----
time->Tue Jan 19 14:54:36 2016
type=AVC msg=audit(1453211676.627:618): avc:  denied  { read } for  pid=7201 comm="systemd" name="libselinux.so.1" dev="dm-1" ino=541346 scontext=system_u:system_r:svirt_lxc_net_t:s0:c9,c955 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0

So it is systemd inside the container trying to read libselinux.so.1 inside the container. ls -liZ returns:

541346 -rwxr-xr-x. 1 root root unconfined_u:object_r:var_lib_t:s0                145440 Sep  9 07:44 /var/lib/rkt/cas/tree/deps-sha512-f5143c3cd24d997aea9a1c0e4e641f4ed9a44edeb4fdfe231a3b8059475100ca/rootfs/usr/lib64/libselinux.so.1
541346 -rwxr-xr-x. 1 root root system_u:object_r:svirt_sandbox_file_t:s0:c9,c955 145440 Sep  9 07:44 /var/lib/rkt/pods/run/0570eb27-fba5-4a2d-8550-9becf0d9ec70/stage1/rootfs/usr/lib64/libselinux.so.1

They are using the same inode because the first one is the overlay fs lower directory, and the second one is in the overlay fs mount.

[alban]

Same error without using overlay fs:

type=AVC msg=audit(1453212466.826:869): avc:  denied  { read } for  pid=9315 comm="systemd" name="libselinux.so.1" dev="dm-1" ino=541825 scontext=system_u:system_r:svirt_lxc_net_t:s0:c581,c949 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0

541825 -rwxr-xr-x. 1 root root unconfined_u:object_r:var_lib_t:s0 145440 Sep  9 07:44 /var/lib/rkt/pods/run/2787d853-c031-483f-a400-450a7be183d0/stage1/rootfs/usr/lib64/libselinux.so.1

[rhatdan]

You can not use SELinux and Overlay at this time. Well I guess you could label everything under /var/lib/rkt/pods as system_u:object_r:svirt_sandbox_file_t:s0 and it might work better. Not sure how you disable SELinux for rkt process separation, but you might have to. Red Hat Kernel Engineers continue to look to improve OverlayFS with SELinux but nothing so far.

[alban]

@rhatdan I tested as well with rkt run --no-overlay so that rkt does not use overlay fs, but I get the same error, see comment above.

Then, I tried:

$ sudo chcon -R  -u system_u -t svirt_sandbox_file_t /var/lib/rkt/pods
$ sudo ./build-rkt-0.15.0+git/bin/rkt run --debug --insecure-options=image --interactive --no-overlay docker://busybox
(...)
Welcome to Linux!

Initializing machine ID from container UUID.
bind(/run/systemd/notify) failed: Permission denied
Failed to bind private socket: Permission denied
Failed to fully start up daemon: No such file or directory
Failed to add watch on /run/systemd/ask-password: No such file or directory

So it works a bit further but still fails. ausearch -m avc -ts recent has plenty of errors; the first ones are:

----
time->Wed Jan 20 16:03:38 2016
type=AVC msg=audit(1453302218.553:2106): avc:  denied  { write } for  pid=4329 comm="systemd" name="systemd" dev="tmpfs" ino=682171 scontext=system_u:system_r:svirt_lxc_net_
t:s0:c433,c492 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=dir permissive=0
----
time->Wed Jan 20 16:03:38 2016
type=AVC msg=audit(1453302218.553:2107): avc:  denied  { write } for  pid=4329 comm="systemd" name="systemd" dev="tmpfs" ino=682171 scontext=system_u:system_r:svirt_lxc_net_
t:s0:c433,c492 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=dir permissive=0
----
time->Wed Jan 20 16:03:38 2016
type=AVC msg=audit(1453302218.553:2108): avc:  denied  { read write } for  pid=4329 comm="systemd" name="tty" dev="tmpfs" ino=682156 scontext=system_u:system_r:svirt_lxc_net
_t:s0:c433,c492 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=chr_file permissive=0
----

[rhatdan]

This looks like this content "systemd dir" and "tty chr_file" are being created by the user process on /tmp and then somehow being used within the container.

I guess rkt guys will need to figure out what is happening, I have not used rkt.

[jonboulle]

Removing from specific milestone as still dependent on external work.

[sym3tri]

am having this same issue in Fedora 23

[iaguis]

am having this same issue in Fedora 23

The fixes (fedora-selinux/selinux-policy#100 and fedora-selinux/selinux-policy#108) are in Fedora 24. There's one more fix (fedora-selinux/selinux-policy#114) which I'm not sure it's included in 24, but it is in Rawhide.

[alban]

The pending kernel patches for SELinux + Overlayfs:
[RFC PATCH 0/9][V3] Overlayfs SELinux Support

/cc @mjg59

[rhatdan]

Yes, it currently looks like this will land in 4.9 kernel.

[jonboulle]

is this resolved?

[rhatdan]

SELinux and Overlayfs are currently working in the Fedora Rawhide Kernel and Fedora 25.

[wenjianhn]

Overlayfs supports selinux since Linux 4.9.

The related commits are the following:

a518b0a selinux: Implement dentry_create_files_as() hook
2602625 security, overlayfs: Provide hook to correctly label newly created
files
c957f6d selinux: Pass security pointer to determine_inode_label()
19472b6 selinux: Implementation for inode_copy_up_xattr() hook
121ab82 security,overlayfs: Provide security hook for copy up of xattrs for
overlay file
56909eb selinux: Implementation for inode_copy_up() hook
d8ad8b4 security, overlayfs: provide copy up security hook for unioned files