Add Fedora / CoreOS 37

[bbaumgartl]

This is a proof of concept which allows RKE to be deployed to SELinux enabled Fedora >= 37

Currently this does not allow kube-apiserver to start because it can't write/append audit logs. Maybe this is related to https://github.com/rancher/rke2-selinux/pull/17/files

[cmurphy]

Thanks! The microos build is failing, I opened another PR to fix it #14

[bbaumgartl]

Do you have an idea on how to allow the kube-apiserver to start again? Maybe something like the workaround from rancher/rke2#692 (comment)?

[cmurphy]

Are you using RKE2 and did you install the rke2-selinux RPM? You may need to add a policy for fedora for that repo. The audit2allow workaround it is not very fine-grained but it should work if installing the rke2-selinux RPM is not an option.

[bbaumgartl]

I am using RKE1. If i install this RPM the selinux check in rke passes but the kube-apiserver can't start up (that's as far as i've gotten). The system audit log shows some failed appends to /opt/rke/var/log/kube-audit/audit-log.json. Maybe it has something to do with the fact that i am using CoreOS and had to set the flex-volume-plugin-dir (https://rke.docs.rancher.com/os#flatcar-container-linux)... i am not sure.

[cmurphy]

What do the selinux audit logs say? There should already be existing rules in this package that apply to /opt/rke, possibly we need to add something special for fedora

https://github.com/rancher/rancher-selinux/pull/13/files#diff-3ca1372da779b1babd4670b9d92963b813c9c5b6ec16698bc148c9f063d8da20R2
https://github.com/rancher/rancher-selinux/pull/13/files#diff-f28bb40da5b5177b25533ad1815737e78522fe8f0d3c0cad1177d4114a6f99c7R62

[bbaumgartl]

Here are some logs i gathered. The kube-apiserver has a bind mount from /opt/rke/var/log/kube-audit to /var/log/kube-audit.

AVC avc:  denied  { write } for  pid=2384 comm="kube-apiserver" name="kube-audit" dev="dm-0" ino=310378725 scontext=system_u:system_r:rke_container_t:s0:c1000,c1001 tcontext=system_u:object_r:var_t:s0 tclass=dir permissive=1
AVC avc:  denied  { remove_name } for  pid=2384 comm="kube-apiserver" name="audit-log.json" dev="dm-0" ino=310388647 scontext=system_u:system_r:rke_container_t:s0:c1000,c1001 tcontext=system_u:object_r:var_t:s0 tclass=dir permissive=1
AVC avc:  denied  { rename } for  pid=2384 comm="kube-apiserver" name="audit-log.json" dev="dm-0" ino=310388647 scontext=system_u:system_r:rke_container_t:s0:c1000,c1001 tcontext=system_u:object_r:var_t:s0 tclass=file permissive=1
AVC avc:  denied  { add_name } for  pid=2384 comm="kube-apiserver" name="audit-log-2023-09-25T06-50-52.519.json" scontext=system_u:system_r:rke_container_t:s0:c1000,c1001 tcontext=system_u:object_r:var_t:s0 tclass=dir permissive=1
AVC avc:  denied  { create } for  pid=2384 comm="kube-apiserver" name="audit-log.json" scontext=system_u:system_r:rke_container_t:s0:c1000,c1001 tcontext=system_u:object_r:var_t:s0 tclass=file permissive=1
AVC avc:  denied  { write open } for  pid=2384 comm="kube-apiserver" path="/var/log/kube-audit/audit-log.json" dev="dm-0" ino=310388650 scontext=system_u:system_r:rke_container_t:s0:c1000,c1001 tcontext=system_u:object_r:var_t:s0 tclass=file permissive=1
AVC avc:  denied  { setattr } for  pid=2384 comm="kube-apiserver" name="audit-log.json" dev="dm-0" ino=310388650 scontext=system_u:system_r:rke_container_t:s0:c1000,c1001 tcontext=system_u:object_r:var_t:s0 tclass=file permissive=1
AVC avc:  denied  { unlink } for  pid=2384 comm="kube-apiserver" name="audit-log-2023-09-25T01-25-41.597.json" dev="dm-0" ino=310388648 scontext=system_u:system_r:rke_container_t:s0:c1000,c1001 tcontext=system_u:object_r:var_t:s0 tclass=file permissive=1
AVC avc:  denied  { nlmsg_read } for  pid=3621523 comm="ss" scontext=system_u:system_r:container_t:s0:c704,c899 tcontext=system_u:system_r:container_t:s0:c704,c899 tclass=netlink_tcpdiag_socket permissive=1
AVC avc:  denied  { read append } for  pid=3715405 comm="kube-apiserver" name="audit-log.json" dev="dm-0" ino=310388650 scontext=system_u:system_r:rke_container_t:s0:c1000,c1001 tcontext=system_u:object_r:var_t:s0 tclass=file permissive=1
AVC avc:  denied  { open } for  pid=3715405 comm="kube-apiserver" path="/var/log/kube-audit/audit-log.json" dev="dm-0" ino=310388650 scontext=system_u:system_r:rke_container_t:s0:c1000,c1001 tcontext=system_u:object_r:var_t:s0 tclass=file permissive=1
AVC avc:  denied  { append } for  pid=3715405 comm="kube-apiserver" name="audit-log.json" dev="dm-0" ino=310388650 scontext=system_u:system_r:rke_container_t:s0:c1000,c1001 tcontext=system_u:object_r:var_t:s0 tclass=file permissive=1
AVC avc:  denied  { open } for  pid=3715405 comm="kube-apiserver" path="/var/log/kube-audit/audit-log.json" dev="dm-0" ino=310388650 scontext=system_u:system_r:rke_container_t:s0:c1000,c1001 tcontext=system_u:object_r:var_t:s0 tclass=file permissive=1
AVC avc:  denied  { nlmsg_read } for  pid=3716568 comm="ss" scontext=system_u:system_r:container_t:s0:c704,c899 tcontext=system_u:system_r:container_t:s0:c704,c899 tclass=netlink_tcpdiag_socket permissive=1
AVC avc:  denied  { getattr } for  pid=1102 comm="systemd-journal" path="/var/log/journal/93c61c9878bc45ea8ae2f4fd749193ee/system@c6fbf2e667ef4716b2c31a7327b851de-0000000004779a2b-00060425cf2a61dd.journal" dev="dm-0" ino=24120671 scontext=system_u:system_r:syslogd_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
AVC avc:  denied  { read } for  pid=3712946 comm="sshd" name="config" dev="dm-0" ino=764424511 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
AVC avc:  denied  { open } for  pid=3712946 comm="sshd" path="/etc/selinux/config" dev="dm-0" ino=764424511 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
AVC avc:  denied  { getattr } for  pid=3712946 comm="sshd" path="/etc/selinux/config" dev="dm-0" ino=764424511 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
AVC avc:  denied  { read } for  pid=3712936 comm="sshd" name="config" dev="dm-0" ino=764424511 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
AVC avc:  denied  { open } for  pid=3712936 comm="sshd" path="/etc/selinux/config" dev="dm-0" ino=764424511 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
AVC avc:  denied  { getattr } for  pid=3712936 comm="sshd" path="/etc/selinux/config" dev="dm-0" ino=764424511 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
AVC avc:  denied  { getattr } for  pid=1102 comm="systemd-journal" path="/var/log/journal/93c61c9878bc45ea8ae2f4fd749193ee/system@c6fbf2e667ef4716b2c31a7327b851de-0000000004779a2b-00060425cf2a61dd.journal" dev="dm-0" ino=24120671 scontext=system_u:system_r:syslogd_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
AVC avc:  denied  { read } for  pid=3712972 comm="systemd-user-ru" name="config" dev="dm-0" ino=764424511 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
AVC avc:  denied  { open } for  pid=3712972 comm="systemd-user-ru" path="/etc/selinux/config" dev="dm-0" ino=764424511 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
AVC avc:  denied  { getattr } for  pid=3712972 comm="systemd-user-ru" path="/etc/selinux/config" dev="dm-0" ino=764424511 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
AVC avc:  denied  { nlmsg_read } for  pid=3713002 comm="ss" scontext=system_u:system_r:container_t:s0:c704,c899 tcontext=system_u:system_r:container_t:s0:c704,c899 tclass=netlink_tcpdiag_socket permissive=1
AVC avc:  denied  { read } for  pid=3713031 comm="systemd-hostnam" name="config" dev="dm-0" ino=764424511 scontext=system_u:system_r:systemd_hostnamed_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
AVC avc:  denied  { open } for  pid=3713031 comm="systemd-hostnam" path="/etc/selinux/config" dev="dm-0" ino=764424511 scontext=system_u:system_r:systemd_hostnamed_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
AVC avc:  denied  { getattr } for  pid=3713031 comm="systemd-hostnam" path="/etc/selinux/config" dev="dm-0" ino=764424511 scontext=system_u:system_r:systemd_hostnamed_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
AVC avc:  denied  { nlmsg_read } for  pid=3713601 comm="ss" scontext=system_u:system_r:container_t:s0:c704,c899 tcontext=system_u:system_r:container_t:s0:c704,c899 tclass=netlink_tcpdiag_socket permissive=1

I think some messages are unrelated to kubernetes but i'm not entirely sure.

[cmurphy]

@bbaumgartl from those logs it looks like the audit log has type var_t, but our type rules are expecting it to be var_log_t. I'm not sure why it's different on your system. You could try changing the file context for /var/log/kube-audit(/.*)? to system_u:object_r:var_log_t:s0, or you could create your own policy using audit2allow or other tools to allow the user rke_container_t to operate on var_t.

[bbaumgartl]

Sorry, it took me a while testing things. While updating the nodes i noticed that one node was fine (selinux in enforcing and kube-apiserver starting). Checking /opt/rke/var/log/kube-audit every file had container_file_t instead of var_t all the other nodes had. I am not sure what or when this was set (it wasn't me). I used this node for testing rke with disabled docker selinux. Maybe rke up changed something while docker selinux was disabled. 🤔

After setting container_file_t on the other nodes (chcon -R -t container_file_t /opt/rke/var/log/kube-audit) everything seems to work fine for now.

Does it make sense to add this workaround (or setting var_log_t on /opt/rke/var/log => does not work) to the documentation?

[dweomer]

Sorry, it took me a while testing things. While updating the nodes i noticed that one node was fine (selinux in enforcing and kube-apiserver starting). Checking /opt/rke/var/log/kube-audit every file had container_file_t instead of var_t all the other nodes had. I am not sure what or when this was set (it wasn't me). I used this node for testing rke with disabled docker selinux. Maybe rke up changed something while docker selinux was disabled. 🤔

After setting container_file_t on the other nodes (chcon -R -t container_file_t /opt/rke/var/log/kube-audit) everything seems to work fine for now.

Does it make sense to add this workaround or setting var_log_t on /opt/rke/var/log to the documentation?

when in doubt, especially with such mixed-mode-over-time scenarios, consider a recursive restorecon on /opt/rke to re-establish baseline assumptions for the runtime+policy.

[bbaumgartl]

A restorecon -R /opt/rke does not seem to have any effect. At least it does not change/reset any types in /opt/rke/var/log/kube-audit regardless of which type is currently set. Maybe it only resets packaged files?

[superseb]

@andypitcher let me know if you need RKE1 team help on this

[macedogm]

@bbaumgartl thanks for submitting this PR. FYI we will review and, if applicable, merge it as soon as we are done with the PR to support EL9 (and tested) and have the build pipelines fixed.

CC @andypitcher