SELinux blocks audit logs by default

[mddamato]

Installing RKE2 in RHEL 8,

[root@rhel-8-0 admin]# rke2 --version
rke2 version v1.19.7+rke2r1 (d1430ad44dc162f15a54d093439c20ffb91f559b)
go version go1.15.5b5
[root@rhel-8-0 admin]# getenforce
Enforcing

/etc/rancher/rke2/config.yaml

selinux: true
profile: cis-1.5

and

/etc/rancher/rke2/audit-policy.yaml:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      # Resource "pods" doesn't match requests to any subresource of pods,
      # which is consistent with the RBAC policy.
      resources: ["pods"]

Once the service is started audit logs do not get populated. I get errors in the Kubernetes-apiserver pods and audit.log

crictl logs:

E0209 15:16:59.030066       1 metrics.go:109] Error in audit plugin 'log' affecting 1 audit events: can't open new logfile: open /var/lib/rancher/rke2/server/logs/audit.log: permission denied
Impacted events:
2021-02-09T15:16:59.029949982Z AUDIT: id="ffe7f2e1-4b30-40cf-82f2-8c2ecd6250e7" stage="RequestReceived" ip="127.0.0.1" method="list" user="system:kube-scheduler" groups="\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/api/v1/pods?limit=500&resourceVersion=0" response="<deferred>"
E0209 15:16:59.030246       1 metrics.go:109] Error in audit plugin 'log' affecting 1 audit events: can't open new logfile: open /var/lib/rancher/rke2/server/logs/audit.log: permission denied
Impacted events:

/var/log/audit/audit.log:

type=PROCTITLE msg=audit(1612883993.173:294): proctitle=6B7562652D617069736572766572002D2D616C6C6F772D70726976696C656765643D74727565002D2D616E6F6E796D6F75732D617574683D66616C7365002D2D6170692D61756469656E6365733D756E6B6E6F776E002D2D617574686F72697A6174696F6E2D6D6F64653D4E6F64652C52424143002D2D62696E642D61646472
type=AVC msg=audit(1612883993.178:295): avc:  denied  { create } for  pid=1795 comm="kube-apiserver" name="audit.log" scontext=system_u:system_r:rke2_service_t:s0:c9,c509 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=file permissive=0
type=SYSCALL msg=audit(1612883993.178:295): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=c01312bbf0 a2=80241 a3=1a4 items=0 ppid=1729 pid=1795 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="kube-apiserver" exe="/usr/local/bin/kube-apiserver" subj=system_u:system_r:rke2_service_t:s0:c9,c509 key=(null)ARCH=x86_64 SYSCALL=openat AUID="unset" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
type=PROCTITLE msg=audit(1612883993.178:295): proctitle=6B7562652D617069736572766572002D2D616C6C6F772D70726976696C656765643D74727565002D2D616E6F6E796D6F75732D617574683D66616C7365002D2D6170692D61756469656E6365733D756E6B6E6F776E002D2D617574686F72697A6174696F6E2D6D6F64653D4E6F64652C52424143002D2D62696E642D61646472

Workaround I've done is using audit2allow to create a policy to allow:

[root@rhel-8-0 admin]# audit2allow -a
#============= rke2_service_t ==============
allow rke2_service_t container_var_lib_t:file create;
allow rke2_service_t container_var_lib_t:file { append rename };

[cjellick]

@dweomer lets see if we can squuze this into 1.20.3 for RKE2. We aren't releasing rke2 1.20 until late feb, so you have most of next week to fit this in.

[davidnuzik]

Added working label as it's being actively worked on.

[davidnuzik]

While this most likely is just tied to an selinux policy rpm release (and not necessarily an RKE2 release) I'm targeting a fix for this by Mid-May (hence the 1.20.7 milestone). We will attempt to get this in by mid-may.
Won't block release (even if ends up needing changes that are not just in the selinux RPM). We'll work to fix this as soon as we are able to.

[briandowns]

Resolve in master of rke2-selinux via PR rancher/rke2-selinux#17

[ShylajaDevadiga]

Validated in rke2 version v1.21.1-rc1+rke2r1 on Centos 8.2 with selinux enabled cis enabled cluster
cat /etc/rancher/rke2/config.yaml

...
profile: cis-1.5
selinux: true

With Policy set to log event metadata, request and response bodies

cat /etc/rancher/rke2/audit-policy.yaml 

apiVersion: audit.k8s.io/v1
kind: Policy
metadata:
  creationTimestamp: null
rules:
- level: RequestResponse

sudo ls /var/lib/rancher/rke2/server/logs/
audit.log

Snippet from the logs

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"8981457a-4bd7-4ca3-89bb-0c9fdb7c6e85","stage":"RequestReceived","requestURI":"/api/v1/namespaces/default/pods/audit-nginx/status","verb":"patch","user":{"username":"system:node:ip-172-31-4-215.us-east-2.compute.internal","groups":["system:nodes","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kubelet/v1.21.1 (linux/amd64) kubernetes/5e58841","objectRef":{"resource":"pods","namespace":"default","name":"audit-nginx","apiVersion":"v1","subresource":"status"},"requestReceivedTimestamp":"2021-05-25T14:24:01.546127Z","stageTimestamp":"2021-05-25T14:24:01.546127Z"}

[brandond]

Tagged v0.7.testing.1 in rke2-selinux - will retag in other channels shortly.