charts: add allowPrivilegeEscalation: true to containerSecurityContext to nodeplugin daemonset #2993

losil · 2022-04-06T05:45:04Z

Describe what this PR does

When running the kubernetes cluster with one single privileged PodSecurityPolicy which is allowing everything the nodeplugin daemonset can fail to start. To be precise the problem is the defaultAllowPrivilegeEscalation: false configuration in the PSP. Containers of the nodeplugin daemonset won't start when they have privileged: true but no allowPrivilegeEscalation in their container securityContext.

Kubernetes will not schedule if this mismatch exists cannot set allowPrivilegeEscalation to false and privileged to true:

[Warning  FailedCreate  7s (x3 over 14s)  daemonset-controller  (combined from similar events): Error creating: Pod "ceph-csi-nodeplugin-rbd-7qm9f" is invalid: [spec.containers[0].securityContext: Invalid value: core.SecurityContext{Capabilities:(*core.Capabilities)(nil), Privileged:(*bool)(0xc04f579a0c), SELinuxOptions:(*core.SELinuxOptions)(nil), WindowsOptions:(*core.WindowsSecurityContextOptions)(nil), RunAsUser:(*int64)(nil), RunAsGroup:(*int64)(nil), RunAsNonRoot:(*bool)(nil), ReadOnlyRootFilesystem:(*bool)(nil), AllowPrivilegeEscalation:(*bool)(0xc04d20ce50), ProcMount:(*core.ProcMountType)(nil), SeccompProfile:(*core.SeccompProfile)(nil)}: cannot set `allowPrivilegeEscalation` to false and `privileged` to true, spec.containers[2].securityContext: Invalid value: core.SecurityContext{Capabilities:(*core.Capabilities)(nil), Privileged:(*bool)(0xc04f579a0f), SELinuxOptions:(*core.SELinuxOptions)(nil), WindowsOptions:(*core.WindowsSecurityContextOptions)(nil), RunAsUser:(*int64)(nil), RunAsGroup:(*int64)(nil), RunAsNonRoot:(*bool)(nil), ReadOnlyRootFilesystem:(*bool)(nil), AllowPrivilegeEscalation:(*bool)(0xc04d20ce50), ProcMount:(*core.ProcMountType)(nil), SeccompProfile:(*core.SeccompProfile)(nil)}: cannot set `allowPrivilegeEscalation` to false and `privileged` to true]]()

The default PodSecurityPolicy for every workload in the k8s cluster is the following:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
  name: system-unrestricted-psp
spec:
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - '*'
  defaultAllowPrivilegeEscalation: false
  fsGroup:
    rule: RunAsAny
  hostIPC: true
  hostNetwork: true
  hostPID: true
  hostPorts:
  - max: 65535
    min: 0
  privileged: true
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - '*'

The daemonsets run when the allowPrivilegeEscalation parameter is added to the container securityContext config. In detail this is needed for the containers:

driver-registrar
liveness-prometheus

Updating the helm chart templates for ceph-csi-rbd and ceph-csi-cephfssolve the problem when using defaultAllowPrivilegeEscalation: true in global privileged PodSecurityPolicy. For example:

{{- if .Values.nodeplugin.httpMetrics.enabled }}
        - name: liveness-prometheus
          securityContext:
            privileged: true
            allowPrivilegeEscalation: true
          image: "{{ .Values.nodeplugin.plugin.image.repository }}:{{ .Values.nodeplugin.plugin.image.tag }}"

Is there anything that requires special attention

Do you have any questions? No

Is the change backward compatible? Yes

Are there concerns around backward compatibility? No

Provide any external context for the change, if any.

For example:

https://kubernetes.io/docs/tasks/configure-pod-container/security-context/

Related issues

Mention any github issues relevant to this PR. Adding below line
will help to auto close the issue once the PR is merged.

None

Future concerns

List items that are not part of the PR and do not impact it's
functionality, but are work items that can be taken up subsequently.

None

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

/retest ci/centos/<job-name>: retest the <job-name> after unrelated
failure (please report the failure too!)
/retest all: run this in case the CentOS CI failed to start/report any test
progress or results

yati1998 · 2022-04-06T06:21:45Z

@losil please check the DCO failure. We have specific guidelines for the commit messages.

humblec · 2022-04-06T08:27:21Z

@losil as you know PSP ( PodSecurityPolicy) is getting deprecated in upstream. Regardless we could have this in chart 👍

losil · 2022-04-06T08:52:47Z

@losil as you know PSP ( PodSecurityPolicy) is getting deprecated in upstream. Regardless we could have this in chart 👍

@humblec of course I am aware of the PSPs deprecation and honestly look forward to getting rid of them.
But since we are still running our clusters with k8s 1.21.10 we still rely on the PSPs.

humblec · 2022-04-06T10:33:17Z

Yeah , please correct the commit lint error @losil .. it should be good to go..

Madhu-1 · 2022-04-06T14:02:30Z

@losil Please check https://github.com/ceph/ceph-csi/blob/devel/docs/development-guide.md#commit-messages standard followed for commit message.

it should be something like below

helm: allowPrivilegeEscalation: true in containerSecurityContext

When running the kubernetes cluster with one single privileged 
PodSecurityPolicy which is allowing everything the nodeplugin 
daemonset can fail to start. To be precise the problem is the 
defaultAllowPrivilegeEscalation: false configuration in the PSP.
 Containers of the nodeplugin daemonset won't start when they 
have privileged: true but no allowPrivilegeEscalation in their c
ontainer securityContext.

Kubernetes will not schedule if this mismatch exists cannot set 
allowPrivilegeEscalation to false and privileged to true:

Signed-off-by: Silvan Loser <[email protected]>

losil · 2022-04-06T15:23:27Z

@losil Please check https://github.com/ceph/ceph-csi/blob/devel/docs/development-guide.md#commit-messages standard followed for commit message.

it should be something like below

helm: allowPrivilegeEscalation: true in containerSecurityContext

When running the kubernetes cluster with one single privileged 
PodSecurityPolicy which is allowing everything the nodeplugin 
daemonset can fail to start. To be precise the problem is the 
defaultAllowPrivilegeEscalation: false configuration in the PSP.
 Containers of the nodeplugin daemonset won't start when they 
have privileged: true but no allowPrivilegeEscalation in their c
ontainer securityContext.

Kubernetes will not schedule if this mismatch exists cannot set 
allowPrivilegeEscalation to false and privileged to true:

Signed-off-by: Silvan Loser <[email protected]>

Thanks for the hint!

humblec · 2022-04-07T04:37:20Z

/retest ci/centos/k8s-e2e-external-storage/1.21

humblec · 2022-04-07T04:37:33Z

/retest ci/centos/mini-e2e-helm/k8s-1.21

humblec · 2022-04-07T10:46:44Z

@Mergifyio rebase

mergify · 2022-04-07T10:47:29Z

rebase

✅ Branch has been successfully rebased

humblec · 2022-04-11T06:41:28Z

@Mergifyio rebase

humblec · 2022-04-11T06:41:37Z

@Mergifyio refresh

mergify · 2022-04-11T06:42:08Z

rebase

✅ Branch has been successfully rebased

mergify · 2022-04-11T06:42:09Z

refresh

✅ Pull request refreshed

Madhu-1

@losil Sorry for the late review on this one.

Same changes are required in the below files

deploy/cephfs/kubernetes/csi-cephfsplugin.yaml
deploy/rbd/kubernetes/csi-rbdplugin.yaml

Pull request has been modified.

Madhu-1 · 2022-04-12T05:59:49Z

@losil can you please change Signed-off-by: Silvan Loser <[email protected]> to Signed-off-by: Silvan Loser <[email protected]> in your 2nd commit?

Pull request has been modified.

ceph-csi-bot · 2022-04-22T15:37:59Z

@losil "ci/centos/mini-e2e-helm/k8s-1.23" test failed. Logs are available at location for debugging

ceph-csi-bot · 2022-04-22T15:38:00Z

/retest ci/centos/mini-e2e/k8s-1.23

ceph-csi-bot · 2022-04-22T15:38:03Z

@losil "ci/centos/mini-e2e/k8s-1.23" test failed. Logs are available at location for debugging

ceph-csi-bot · 2022-04-22T15:38:03Z

@Mergifyio requeue

mergify · 2022-04-22T15:38:17Z

requeue

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

ceph-csi-bot · 2022-04-22T16:41:24Z

/retest ci/centos/mini-e2e-helm/k8s-1.21

ceph-csi-bot · 2022-04-22T16:41:25Z

@losil "ci/centos/mini-e2e-helm/k8s-1.21" test failed. Logs are available at location for debugging

ceph-csi-bot · 2022-04-22T16:41:26Z

/retest ci/centos/k8s-e2e-external-storage/1.22

ceph-csi-bot · 2022-04-22T16:41:27Z

@losil "ci/centos/k8s-e2e-external-storage/1.22" test failed. Logs are available at location for debugging

ceph-csi-bot · 2022-04-22T16:41:28Z

@Mergifyio requeue

mergify · 2022-04-22T16:41:57Z

requeue

☑️ This pull request is already queued

ceph-csi-bot · 2022-04-22T17:38:03Z

/retest ci/centos/mini-e2e-helm/k8s-1.21

ceph-csi-bot · 2022-04-22T17:38:04Z

@losil "ci/centos/mini-e2e-helm/k8s-1.21" test failed. Logs are available at location for debugging

ceph-csi-bot · 2022-04-22T17:38:05Z

/retest ci/centos/mini-e2e-helm/k8s-1.22

ceph-csi-bot · 2022-04-22T17:38:06Z

@losil "ci/centos/mini-e2e-helm/k8s-1.22" test failed. Logs are available at location for debugging

ceph-csi-bot · 2022-04-22T17:38:06Z

@Mergifyio requeue

mergify · 2022-04-22T17:38:35Z

requeue

☑️ This pull request is already queued

ceph-csi-bot · 2022-04-22T19:31:45Z

/retest ci/centos/mini-e2e-helm/k8s-1.22

ceph-csi-bot · 2022-04-22T19:31:47Z

@losil "ci/centos/mini-e2e-helm/k8s-1.22" test failed. Logs are available at location for debugging

ceph-csi-bot · 2022-04-22T19:31:48Z

@Mergifyio requeue

mergify · 2022-04-22T19:32:18Z

requeue

☑️ This pull request is already queued

ceph-csi-bot · 2022-04-22T21:33:13Z

/retest ci/centos/mini-e2e-helm/k8s-1.22

ceph-csi-bot · 2022-04-22T21:33:13Z

@losil "ci/centos/mini-e2e-helm/k8s-1.22" test failed. Logs are available at location for debugging

ceph-csi-bot · 2022-04-22T21:33:14Z

@Mergifyio requeue

mergify · 2022-04-22T21:33:50Z

requeue

☑️ This pull request is already queued

Madhu-1 · 2022-04-26T05:15:08Z

@Mergifyio refresh

mergify · 2022-04-26T05:15:28Z

refresh

✅ Pull request refreshed

losil force-pushed the fix/container-security-context branch from fdf49dc to c46fdcf Compare April 6, 2022 06:28

losil changed the title ~~add allowPrivilegeEscalation: true to containerSecurityContext to nodeplugin daemonset~~ charts: add allowPrivilegeEscalation: true to containerSecurityContext to nodeplugin daemonset Apr 6, 2022

losil force-pushed the fix/container-security-context branch from c46fdcf to 1f8ff5a Compare April 6, 2022 11:09

losil force-pushed the fix/container-security-context branch from 1f8ff5a to 413d283 Compare April 6, 2022 15:23

humblec previously approved these changes Apr 7, 2022

View reviewed changes

humblec added the backport-to-release-v3.6 label Apr 7, 2022

yati1998 previously approved these changes Apr 7, 2022

View reviewed changes

Rakshith-R force-pushed the fix/container-security-context branch from 413d283 to eae27dc Compare April 7, 2022 10:47

Rakshith-R force-pushed the fix/container-security-context branch from eae27dc to 03dae1b Compare April 11, 2022 06:42

Madhu-1 requested changes Apr 11, 2022

View reviewed changes

Madhu-1 previously approved these changes Apr 12, 2022

View reviewed changes

losil force-pushed the fix/container-security-context branch from bce1ff2 to bff4f29 Compare April 12, 2022 07:56

mergify bot merged commit f2e0fa2 into ceph:devel Apr 22, 2022

Madhu-1 added the backport-to-release-v3.6 label Apr 26, 2022

mergify bot mentioned this pull request Apr 26, 2022

charts: add allowPrivilegeEscalation: true to containerSecurityContext to nodeplugin daemonset (backport #2993) #3065

Merged

charts: add allowPrivilegeEscalation: true to containerSecurityContext to nodeplugin daemonset #2993

charts: add allowPrivilegeEscalation: true to containerSecurityContext to nodeplugin daemonset #2993

Conversation

losil commented Apr 6, 2022

Describe what this PR does

Is there anything that requires special attention

Related issues

Future concerns

yati1998 commented Apr 6, 2022

humblec commented Apr 6, 2022 • edited

losil commented Apr 6, 2022

humblec commented Apr 6, 2022

Madhu-1 commented Apr 6, 2022

losil commented Apr 6, 2022

humblec commented Apr 7, 2022

humblec commented Apr 7, 2022

humblec commented Apr 7, 2022

mergify bot commented Apr 7, 2022

✅ Branch has been successfully rebased

humblec commented Apr 11, 2022

humblec commented Apr 11, 2022

mergify bot commented Apr 11, 2022

✅ Branch has been successfully rebased

mergify bot commented Apr 11, 2022

✅ Pull request refreshed

Madhu-1 left a comment

Choose a reason for hiding this comment

Madhu-1 commented Apr 12, 2022

ceph-csi-bot commented Apr 22, 2022

ceph-csi-bot commented Apr 22, 2022

ceph-csi-bot commented Apr 22, 2022

ceph-csi-bot commented Apr 22, 2022

mergify bot commented Apr 22, 2022

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

ceph-csi-bot commented Apr 22, 2022

ceph-csi-bot commented Apr 22, 2022

ceph-csi-bot commented Apr 22, 2022

ceph-csi-bot commented Apr 22, 2022

ceph-csi-bot commented Apr 22, 2022

mergify bot commented Apr 22, 2022

☑️ This pull request is already queued

ceph-csi-bot commented Apr 22, 2022

ceph-csi-bot commented Apr 22, 2022

ceph-csi-bot commented Apr 22, 2022

ceph-csi-bot commented Apr 22, 2022

ceph-csi-bot commented Apr 22, 2022

mergify bot commented Apr 22, 2022

☑️ This pull request is already queued

ceph-csi-bot commented Apr 22, 2022

ceph-csi-bot commented Apr 22, 2022

ceph-csi-bot commented Apr 22, 2022

mergify bot commented Apr 22, 2022

☑️ This pull request is already queued

ceph-csi-bot commented Apr 22, 2022

ceph-csi-bot commented Apr 22, 2022

ceph-csi-bot commented Apr 22, 2022

mergify bot commented Apr 22, 2022

☑️ This pull request is already queued

Madhu-1 commented Apr 26, 2022

mergify bot commented Apr 26, 2022

✅ Pull request refreshed

humblec commented Apr 6, 2022 •

edited