Mount Pod stuck in Pending state: PVC is not bound

[Exubient]

What happened:

• I'm trying to make JuiceFS work on a EKS-s3 setup.

• I have managed to make the PoC application work successfully using a dynamic provisioning/ using StorageClass.

• After having tested the whole setup works, I tried changing the configs so that the cache for the mount pod would run on a seperate PVC(EBS in my case). option: juicefs/mount-cache-pvc: XXX

• Thus problem when doing so, the mount pod is stuck in a pending state with the error:
  Unable to attach or mount volumes: unmounted volumes=[cachedir-pvc-0], unattached volumes=[], failed to process volumes=[cachedir-pvc-0]: error processing PVC kube-system/{NAME_OF_CACHE}: PVC is not bound

• and the pvc stuck in state:
  waiting for first consumer to be created before binding

What you expected to happen:

• I've manually created the PVC, and know that it works find when attaching to normal pods. The PV is created and a EBS volume is created on my AWS console.
• I would expect the same to happen when attaching the PVC to the juicefs mount pod.

How to reproduce it (as minimally and precisely as possible):

• Dynamic Provisioning.
• StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: my-fjs
provisioner: csi.juicefs.com
reclaimPolicy: Delete
allowVolumeExpansion: true
parameters:
  csi.storage.k8s.io/provisioner-secret-name: fjs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: default
  csi.storage.k8s.io/node-publish-secret-name: fjs-secret
  csi.storage.k8s.io/node-publish-secret-namespace: default
  juicefs/mount-cpu-limit: 400m
  juicefs/mount-memory-limit: 1000Mi
  juicefs/mount-cpu-request: 400m
  juicefs/mount-memory-request: 1000Mi
  juicefs/mount-delete-delay: 10m
  juicefs/mount-cache-pvc: "my-cache-pvc"
mountOptions:
  - cache-dir=/var/jfsCache
  - cache-size=204800

• Secret:

apiVersion: v1
kind: Secret
metadata:
  name: fjs-secret
type: Opaque
stringData:
  name: jfs
  metaurl: {REDIS} <- this works fine
  storage: s3
  bucket: {SOME_BUCKET}
  access-key: {KEY} 
  secret-key: {SECRET}

• PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-cache-pvc
  namespace: kube-system
status:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 40Gi
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 40Gi
  storageClassName: gp3 <- this gp3 setup has no problem
  volumeMode: Filesystem

• application:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-apache-0-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: my-fjs
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-apache-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-apache-1
  template:
    metadata:
      labels:
        app: my-apache-1
    spec:
      containers:
      - name: apache
        image: httpd:latest
        volumeMounts:
        - name: data
          mountPath: /usr/local/apache2/htdocs
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: my-apache-0-pvc

Mount Pod details

apiVersion: v1
kind: Pod
metadata:
  name: >-
    juicefs-ip-{REDACTED}-pvc-ab7baaf4-c582-42a5-a4a8-0de042fdbedb-amqjzk
  namespace: kube-system
  uid: 9cedafb1-90c1-468b-9f52-3eca6a07f1a0
  resourceVersion: '334509614'
  creationTimestamp: '2024-03-08T08:08:30Z'
  labels:
    app.kubernetes.io/name: juicefs-mount
    juicefs-hash: 0cf88f75cfece633d2d16ba0c4492c9531e5cdb1004d620bce143c22770a12a
    volume-id: pvc-ab7baaf4-c582-42a5-a4a8-0de042fdbedb
  annotations:
    juicefs-8fed6b072a2c61837bdb2f2f80858f883987db176d04483e1a3945f: >-
      /var/lib/kubelet/pods/a3d020e8-95b3-4efb-be5c-4244e330bf21/volumes/kubernetes.io~csi/pvc-ab7baaf4-c582-42a5-a4a8-0de042fdbedb/mount
    juicefs-delete-delay: 10m
    juicefs-uniqueid: pvc-ab7baaf4-c582-42a5-a4a8-0de042fdbedb
    juicefs-uuid: ''
  finalizers:
    - juicefs.com/finalizer
  managedFields:
    - manager: juicefs-csi-driver
      operation: Update
      apiVersion: v1
      time: '2024-03-08T08:08:30Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:juicefs-8fed6b072a2c61837bdb2f2f80858f883987db176d04483e1a3945f: {}
            f:juicefs-delete-delay: {}
            f:juicefs-uniqueid: {}
            f:juicefs-uuid: {}
          f:finalizers:
            .: {}
            v:"juicefs.com/finalizer": {}
          f:labels:
            .: {}
            f:app.kubernetes.io/name: {}
            f:juicefs-hash: {}
            f:volume-id: {}
        f:spec:
          f:containers:
            k:{"name":"jfs-mount"}:
              .: {}
              f:command: {}
              f:env:
                .: {}
                k:{"name":"JFS_FOREGROUND"}:
                  .: {}
                  f:name: {}
                  f:value: {}
              f:envFrom: {}
              f:image: {}
              f:imagePullPolicy: {}
              f:lifecycle:
                .: {}
                f:preStop:
                  .: {}
                  f:exec:
                    .: {}
                    f:command: {}
              f:name: {}
              f:ports:
                .: {}
                k:{"containerPort":9567,"protocol":"TCP"}:
                  .: {}
                  f:containerPort: {}
                  f:name: {}
                  f:protocol: {}
              f:resources:
                .: {}
                f:limits:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
                f:requests:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
              f:securityContext:
                .: {}
                f:privileged: {}
                f:runAsUser: {}
              f:terminationMessagePath: {}
              f:terminationMessagePolicy: {}
              f:volumeMounts:
                .: {}
                k:{"mountPath":"/etc/updatedb.conf"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/jfs"}:
                  .: {}
                  f:mountPath: {}
                  f:mountPropagation: {}
                  f:name: {}
                k:{"mountPath":"/var/jfsCache"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/var/jfsCache-0"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
          f:dnsPolicy: {}
          f:enableServiceLinks: {}
          f:nodeName: {}
          f:preemptionPolicy: {}
          f:priorityClassName: {}
          f:restartPolicy: {}
          f:schedulerName: {}
          f:securityContext: {}
          f:serviceAccount: {}
          f:serviceAccountName: {}
          f:terminationGracePeriodSeconds: {}
          f:tolerations: {}
          f:volumes:
            .: {}
            k:{"name":"cachedir-0"}:
              .: {}
              f:hostPath:
                .: {}
                f:path: {}
                f:type: {}
              f:name: {}
            k:{"name":"cachedir-pvc-0"}:
              .: {}
              f:name: {}
              f:persistentVolumeClaim:
                .: {}
                f:claimName: {}
            k:{"name":"jfs-dir"}:
              .: {}
              f:hostPath:
                .: {}
                f:path: {}
                f:type: {}
              f:name: {}
            k:{"name":"updatedb"}:
              .: {}
              f:hostPath:
                .: {}
                f:path: {}
                f:type: {}
              f:name: {}
    - manager: kubelet
      operation: Update
      apiVersion: v1
      time: '2024-03-08T08:08:30Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:conditions:
            .: {}
            k:{"type":"ContainersReady"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
            k:{"type":"Initialized"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
            k:{"type":"PodScheduled"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
            k:{"type":"Ready"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
          f:containerStatuses: {}
          f:hostIP: {}
          f:startTime: {}
      subresource: status
  selfLink: >-
    /api/v1/namespaces/kube-system/pods/juicefs-{REDACTED}-pvc-ab7baaf4-c582-42a5-a4a8-0de042fdbedb-amqjzk
status:
  phase: Pending
  conditions:
    - type: Initialized
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2024-03-08T08:08:30Z'
    - type: Ready
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2024-03-08T08:08:30Z'
      reason: ContainersNotReady
      message: 'containers with unready status: [jfs-mount]'
    - type: ContainersReady
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2024-03-08T08:08:30Z'
      reason: ContainersNotReady
      message: 'containers with unready status: [jfs-mount]'
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2024-03-08T08:08:30Z'
  hostIP: 10.0.27.65
  startTime: '2024-03-08T08:08:30Z'
  containerStatuses:
    - name: jfs-mount
      state:
        waiting:
          reason: ContainerCreating
      lastState: {}
      ready: false
      restartCount: 0
      image: juicedata/mount:ce-v1.1.1
      imageID: ''
      started: false
  qosClass: Guaranteed
spec:
  volumes:
    - name: jfs-dir
      hostPath:
        path: /var/lib/juicefs/volume
        type: DirectoryOrCreate
    - name: updatedb
      hostPath:
        path: /etc/updatedb.conf
        type: FileOrCreate
    - name: cachedir-0
      hostPath:
        path: /var/jfsCache
        type: DirectoryOrCreate
    - name: cachedir-pvc-0
      persistentVolumeClaim:
        claimName: my-cache-pvc
    - name: kube-api-access-ph6zj
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
        defaultMode: 420
  containers:
    - name: jfs-mount
      image: juicedata/mount:ce-v1.1.1
      command:
        - sh
        - '-c'
        - >-
          /usr/local/bin/juicefs format --storage=s3
          --bucket={REDACTED}
          --access-key={REDACTED} --secret-key=${secretkey} ${metaurl}
          {projectname}

          /bin/mount.juicefs ${metaurl}
          /jfs/pvc-ab7baaf4-c582-42a5-a4a8-0de042fdbedb-amqjzk -o
          cache-size=204800,cache-dir=/var/jfsCache-0:/var/jfsCache,metrics=0.0.0.0:9567
      ports:
        - name: metrics
          containerPort: 9567
          protocol: TCP
      envFrom:
        - secretRef:
            name: >-
              juicefs-ip-{REDACTED}-pvc-ab7baaf4-c582-42a5-a4a8-0de042fdbedb-amqjzk-secret
      env:
        - name: JFS_FOREGROUND
          value: '1'
      resources:
        limits:
          cpu: 400m
          memory: 1000Mi
        requests:
          cpu: 400m
          memory: 1000Mi
      volumeMounts:
        - name: jfs-dir
          mountPath: /jfs
          mountPropagation: Bidirectional
        - name: updatedb
          mountPath: /etc/updatedb.conf
        - name: cachedir-0
          mountPath: /var/jfsCache
        - name: cachedir-pvc-0
          mountPath: /var/jfsCache-0
        - name: kube-api-access-ph6zj
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      lifecycle:
        preStop:
          exec:
            command:
              - sh
              - '-c'
              - >-
                umount /jfs/pvc-ab7baaf4-c582-42a5-a4a8-0de042fdbedb-amqjzk -l
                && rmdir /jfs/pvc-ab7baaf4-c582-42a5-a4a8-0de042fdbedb-amqjzk
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext:
        privileged: true
        runAsUser: 0
  restartPolicy: Always
  terminationGracePeriodSeconds: 10
  dnsPolicy: ClusterFirstWithHostNet
  serviceAccountName: juicefs-csi-node-sa
  serviceAccount: juicefs-csi-node-sa
  nodeName: {REDACTED}
  securityContext: {}
  schedulerName: default-scheduler
  tolerations:
    - key: CriticalAddonsOnly
      operator: Exists
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
    - key: node.kubernetes.io/disk-pressure
      operator: Exists
      effect: NoSchedule
    - key: node.kubernetes.io/memory-pressure
      operator: Exists
      effect: NoSchedule
    - key: node.kubernetes.io/pid-pressure
      operator: Exists
      effect: NoSchedule
    - key: node.kubernetes.io/unschedulable
      operator: Exists
      effect: NoSchedule
  priorityClassName: system-node-critical
  priority: 2000001000
  enableServiceLinks: true
  preemptionPolicy: PreemptLowerPriority

Anything else we need to know?

• I'd like to point out that without juicefs/mount-cache-pvc: "my-cache-pvc", the above settings works fine.
• I'm really curious why PV is not created by the PVC from the juiceFS mount pod (in namespace: kube-system)

Environment:

• JuiceFS version (use juicefs --version) or Hadoop Java SDK version: Using image juicedata/mount:ce-v1.1.1
• Cloud provider or hardware configuration running JuiceFS: AWS
• OS (e.g cat /etc/os-release):
• Kernel (e.g. uname -a):
• Object storage (cloud provider and region, or self maintained): S3
• Metadata engine info (version, cloud provider managed or self maintained):
• Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage):
• Others:
  • All the above I'm not able to access since I'm not able to debug and enter the mount pod (since cache volume is not being mounted)

Thank you!

[showjason]

Hi @Exubient, it seems like the status of cache pvc my-cache-pvc is not bound, which results in the mount pod can't find my-cache-pvc. In my opinion, you should address the issue of cache pvc firstly.

[Exubient]

Hi @showjason,
could you please elaborate on what you mean by address the issue of cache pvc?
From documentation, I don't see where the above setup could have gone wrong..

[zxh326]

plz check my-cache-pvc created first and was bound.

You may need to confirm the gp3 storageClass is not working reason...

[showjason]

Hi @showjason, could you please elaborate on what you mean by address the issue of cache pvc? From documentation, I don't see where the above setup could have gone wrong..

I mean you should firstly make sure the cache pvc my-cache-pvc is created and works properly.
The error message PVC is not bound indicates the status of my-cache-pvc is not bound, you need to find the reason.
Perhaps, the storageClass gp3 can't create pv as expected or something else, but not caused by juicefs-csi-driver, you need to figure it out.

[Exubient]

Thanks for the input, but i would like to add that gp3 is working fine as the current setting.
for example, I've just confirmed that with the following two files the pvc is correctly mounted and the EBS volume is created.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: some-working-pvc
status:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 40Gi
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 40Gi
  storageClassName: gp3
  volumeMode: Filesystem
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-apache-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-apache-1
  template:
    metadata:
      labels:
        app: my-apache-1
    spec:
      containers:
      - name: apache
        image: httpd:latest
        volumeMounts:
        - name: data
          mountPath: /usr/local/apache2/htdocs
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: some-working-pvc

[showjason]

Thanks for the input, but i would like to add that gp3 is working fine as the current setting. for example, I've just confirmed that with the following two files the pvc is correctly mounted and the EBS volume is created.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: some-working-pvc
status:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 40Gi
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 40Gi
  storageClassName: gp3
  volumeMode: Filesystem
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-apache-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-apache-1
  template:
    metadata:
      labels:
        app: my-apache-1
    spec:
      containers:
      - name: apache
        image: httpd:latest
        volumeMounts:
        - name: data
          mountPath: /usr/local/apache2/htdocs
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: some-working-pvc

The deployment order is:

1. Create the cache pvc my-cache-pvc in namespace kube-system, confirm the my-cache-pvc is created successfully and status must be bound. You said the pvc stuck in state waiting for first consumer to be created before binding, you must find the root cause and fix it.
2. Deploy the application.

My suggestion is deploying the cache pvc again and checking its status, if bound, deploy the application, if not bound, fix it, then deploy the application.

[Exubient]

@showjason
thank you once more for the input.

It's quite frustrating for me to debug this issue for the following reasons.

1. I'm unable to enter the my pod juicefs-ip-10-0-44-104-pvc-c6b52a01-70aa-44a0-9594-5f8900ff88a6-stgvzy as the mount failed.
2. From my perspective, all my other pods/pvcs that use storageClass gp3 works fine and in well bound to the pod. Just this one using by the mount pod juicefs-ip-10-0-44-104-pvc-c6b52a01-70aa-44a0-9594-5f8900ff88a6-stgvzy

3. when I run kubectl describe pvc my-cache-pvc -n kube-system, WaitForFirstConsumer does not make sense since it's already Used by the mount pod like the below.

Name:          my-cache-pvc
Namespace:     kube-system
StorageClass:  gp3
Status:        Pending
Volume:
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       juicefs-ip-10-0-44-104-pvc-c6b52a01-70aa-44a0-9594-5f8900ff88a6-stgvzy
Events:
  Type    Reason                Age                     From                         Message
  ----    ------                ----                    ----                         -------
  Normal  WaitForFirstConsumer  2m13s (x26 over 8m17s)  persistentvolume-controller  waiting for first consumer to be created before binding

[showjason]

@Exubient
try to list the pv with kubectl get pv to check the persistent volume corresponding to cache pvc my-cache-pvc is created or not

[Exubient]

the pv is not created, only the pvc in stuck in the Pending state

[showjason]

try to figure out why this pv is not created, maybe this is due to the resourcequota in namespace kube-system storage resource quota or something else, you need to check the kubernetes cluster.

[chenmiao1991]

我使用静态pv也碰到同样的问题，这个多副本的情况下必现的。因为你的缓存pvc是ReadWriteOnce的。第一个mount pod可以正常应用缓存pvc，第二个mount pod会因为缓存pvc不能多挂而pending

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-cache-pvc
  namespace: kube-system
status:
  accessModes:
    - ReadWriteOnce    <--这里
  capacity:
    storage: 40Gi
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 40Gi
  storageClassName: gp3 <- this gp3 setup has no problem
  volumeMode: Filesystem

[chenmiao1991]

To fix this bug, it is recommended to use volumeClaimTemplates like juicefs/mount-cache-pvcTemplates instead of juicefs/mount-cache-pvc. Hope that the authorities will implement it

......
  volumeClaimTemplates:
  - metadata:
      name: bs-cache-pvc
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
      storageClassName: block-storage

[zxh326]

for this case, changing the volumeBindingMode of the gp3 storageClass to Immediate should be able to solve this problem.

[chenmiao1991]

I open a new issue. Phenomenon is different. #906