Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] 同时在同一个node首次创建同一个sc的pv、pvc并挂载，配置csi参数STORAGE_CLASS_SHARE_MOUNT=true。也会出现在同一个node上启动多个mount pod #1063

Closed
luoxingit opened this issue Aug 1, 2024 · 2 comments · Fixed by #1067
Closed

[BUG] 同时在同一个node首次创建同一个sc的pv、pvc并挂载，配置csi参数STORAGE_CLASS_SHARE_MOUNT=true。也会出现在同一个node上启动多个mount pod #1063

luoxingit opened this issue Aug 1, 2024 · 2 comments · Fixed by #1067
Labels
kind/bug Something isn't working

Comments

@luoxingit
Copy link

895259be15c55743c01d0d4214af960
@luoxingit luoxingit added the kind/bug Something isn't working label Aug 1, 2024
@zwwhdls
Copy link
Member

zwwhdls commented Aug 1, 2024

Can you provide the yaml of the three mount pod?

@luoxingit
Copy link
Author

of course

apiVersion: v1
kind: Pod
metadata:
annotations:
juicefs-a1ddf2e3c180eae277cd26153516c96a2edb59fbc7411315269aa7e: >-
/var/lib/kubelet/pods/d1539627-ab16-4949-ad94-b66ea3612767/volumes/kubernetes.io~csi/pvc-246eb908-2341-4666-839c-19566d4d34ae/mount
juicefs-uniqueid: juicefs-sc
juicefs-uuid: 46d322df-f1cf-473f-b69e-6d67c799a5da
k8s.v1.cni.cncf.io/network-status: |-
[{
"name": "kube-ovn",
"interface": "eth0",
"ips": [
"10.233.1.159"
],
"mac": "9a:4d:45:6a:be:e9",
"default": true,
"dns": {},
"gateway": [
"10.233.0.1"
]
}]
ovn.kubernetes.io/allocated: 'true'
ovn.kubernetes.io/cidr: 10.233.0.0/16
ovn.kubernetes.io/gateway: 10.233.0.1
ovn.kubernetes.io/ip_address: 10.233.1.159
ovn.kubernetes.io/logical_router: ovn-cluster
ovn.kubernetes.io/logical_switch: ovn-default
ovn.kubernetes.io/mac_address: '9a:4d:45:6a:be:e9'
ovn.kubernetes.io/pod_nic_type: veth-pair
ovn.kubernetes.io/routed: 'true'
finalizers:
- juicefs.com/finalizer
labels:
app.kubernetes.io/name: juicefs-mount
juicefs-hash: d2708a3b0b9bd9af86909b9215c5d002d8a51475b1cc13870afabf644a4ad11
volume-id: juicefs-sc
name: juicefs-paas-pre-1.novalocal-juicefs-sc-bjeivh
namespace: paas-controll-juceisfs
resourceVersion: '14163451'
spec:
containers:
- command:
- sh
- '-c'
- >-
/usr/local/bin/juicefs format --storage=s3
--bucket=http://192.168.239.24:8080/paaspre
--access-key=B0273VHX8FO2FGW47ON3 --trash-days=0
--secret-key=${secretkey} ${metaurl} paas-juicefs-default

      /bin/mount.juicefs ${metaurl} /jfs/juicefs-sc-bjeivh -o
      metrics=0.0.0.0:9567
  env:
    - name: JFS_FOREGROUND
      value: '1'
  envFrom:
    - secretRef:
        name: juicefs-paas-pre-1.novalocal-juicefs-sc-bjeivh-secret
  image: 'dockerhub.vertexwan.cn/pub/juicefsmountpod:ce-v1.2.0'
  imagePullPolicy: IfNotPresent
  lifecycle:
    preStop:
      exec:
        command:
          - sh
          - '-c'
          - +e
          - >-
            umount -l /jfs/juicefs-sc-bjeivh; rmdir /jfs/juicefs-sc-bjeivh;
            exit 0
  name: jfs-mount
  ports:
    - containerPort: 9567
      name: metrics
      protocol: TCP
  resources:
    limits:
      cpu: '1'
      memory: 2Gi
    requests:
      cpu: '1'
      memory: 2Gi
  securityContext:
    privileged: true
    runAsUser: 0
  terminationMessagePath: /dev/termination-log
  terminationMessagePolicy: File
  volumeMounts:
    - mountPath: /jfs
      mountPropagation: Bidirectional
      name: jfs-dir
    - mountPath: /etc/updatedb.conf
      name: updatedb
    - mountPath: /var/jfsCache
      name: cachedir-0
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-9q6h7
      readOnly: true

dnsPolicy: ClusterFirstWithHostNet
enableServiceLinks: true
hostname: pvc-246eb908-2341-4666-839c-19566d4d34ae
nodeName: paas-pre-1.novalocal
preemptionPolicy: PreemptLowerPriority
priority: 2000001000
priorityClassName: system-node-critical
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: juicefs-csi-node-sa
serviceAccountName: juicefs-csi-node-sa
terminationGracePeriodSeconds: 10
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/disk-pressure
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/memory-pressure
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/pid-pressure
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/unschedulable
operator: Exists
volumes:
- hostPath:
path: /var/lib/juicefs/volume
type: DirectoryOrCreate
name: jfs-dir
- hostPath:
path: /etc/updatedb.conf
type: FileOrCreate
name: updatedb
- hostPath:
path: /var/jfsCache
type: DirectoryOrCreate
name: cachedir-0
- name: kube-api-access-9q6h7
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: '2024-07-26T07:09:58Z'
status: 'True'
type: Initialized
- lastProbeTime: null
lastTransitionTime: '2024-07-26T07:09:59Z'
status: 'True'
type: Ready
- lastProbeTime: null
lastTransitionTime: '2024-07-26T07:09:59Z'
status: 'True'
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: '2024-07-26T07:09:58Z'
status: 'True'
type: PodScheduled
containerStatuses:
- containerID: >-
containerd://fe93c3af7d2fe9e74096121a8f67b30b510fb24b04439d1c1fe86a3a97f4908e
image: 'docker.io/juicedata/mount:ce-v1.2.0'
imageID: >-
docker.io/juicedata/mount@sha256:b28eb957b79068d16ebc26ec31f1da78086decbe8e450a46ff328ddcad853810
lastState: {}
name: jfs-mount
ready: true
restartCount: 0
started: true
state:
running:
startedAt: '2024-07-26T07:09:59Z'
hostIP: 192.168.239.102
phase: Running
podIP: 10.233.1.159
podIPs:
- ip: 10.233.1.159
qosClass: Guaranteed
startTime: '2024-07-26T07:09:58Z'

apiVersion: v1
kind: Pod
metadata:
annotations:
juicefs-b034c240a7cdf07e2388cfb32ad4db7fca957f7508860bc0e620905: >-
/var/lib/kubelet/pods/43b6f5ef-de1e-4532-ab56-c012bffd1ca4/volumes/kubernetes.io~csi/pvc-5ebee006-b033-48cd-8eda-36cdd0df465b/mount
juicefs-uniqueid: juicefs-sc
juicefs-uuid: 46d322df-f1cf-473f-b69e-6d67c799a5da
k8s.v1.cni.cncf.io/network-status: |-
[{
"name": "kube-ovn",
"interface": "eth0",
"ips": [
"10.233.0.206"
],
"mac": "e2:38:b2:ba:38:5d",
"default": true,
"dns": {},
"gateway": [
"10.233.0.1"
]
}]
ovn.kubernetes.io/allocated: 'true'
ovn.kubernetes.io/cidr: 10.233.0.0/16
ovn.kubernetes.io/gateway: 10.233.0.1
ovn.kubernetes.io/ip_address: 10.233.0.206
ovn.kubernetes.io/logical_router: ovn-cluster
ovn.kubernetes.io/logical_switch: ovn-default
ovn.kubernetes.io/mac_address: 'e2:38:b2:ba:38:5d'
ovn.kubernetes.io/pod_nic_type: veth-pair
ovn.kubernetes.io/routed: 'true'
finalizers:
- juicefs.com/finalizer
labels:
app.kubernetes.io/name: juicefs-mount
juicefs-hash: 522ce254977d78df0aad6a2e286ecc6c1b493151777f41e27a34cee883c6d42
volume-id: juicefs-sc
name: juicefs-paas-pre-1.novalocal-juicefs-sc-rjcyhx
namespace: paas-controll-juceisfs
resourceVersion: '14133391'
spec:
containers:
- command:
- sh
- '-c'
- >-
/usr/local/bin/juicefs format --storage=s3
--bucket=http://192.168.239.24:8080/paaspre
--access-key=B0273VHX8FO2FGW47ON3 --trash-days=1
--secret-key=${secretkey} ${metaurl} paas-juicefs-default

      /bin/mount.juicefs ${metaurl} /jfs/juicefs-sc-rjcyhx -o
      metrics=0.0.0.0:9567
  env:
    - name: JFS_FOREGROUND
      value: '1'
  envFrom:
    - secretRef:
        name: juicefs-paas-pre-1.novalocal-juicefs-sc-rjcyhx-secret
  image: 'dockerhub.vertexwan.cn/pub/juicefsmountpod:ce-v1.2.0'
  imagePullPolicy: IfNotPresent
  lifecycle:
    preStop:
      exec:
        command:
          - sh
          - '-c'
          - +e
          - >-
            umount -l /jfs/juicefs-sc-rjcyhx; rmdir /jfs/juicefs-sc-rjcyhx;
            exit 0
  name: jfs-mount
  ports:
    - containerPort: 9567
      name: metrics
      protocol: TCP
  resources:
    limits:
      cpu: '1'
      memory: 2Gi
    requests:
      cpu: '1'
      memory: 2Gi
  securityContext:
    privileged: true
    runAsUser: 0
  terminationMessagePath: /dev/termination-log
  terminationMessagePolicy: File
  volumeMounts:
    - mountPath: /jfs
      mountPropagation: Bidirectional
      name: jfs-dir
    - mountPath: /etc/updatedb.conf
      name: updatedb
    - mountPath: /var/jfsCache
      name: cachedir-0
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-h8ljl
      readOnly: true

dnsPolicy: ClusterFirstWithHostNet
enableServiceLinks: true
hostname: pvc-5ebee006-b033-48cd-8eda-36cdd0df465b
nodeName: paas-pre-1.novalocal
preemptionPolicy: PreemptLowerPriority
priority: 2000001000
priorityClassName: system-node-critical
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: juicefs-csi-node-sa
serviceAccountName: juicefs-csi-node-sa
terminationGracePeriodSeconds: 10
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/disk-pressure
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/memory-pressure
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/pid-pressure
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/unschedulable
operator: Exists
volumes:
- hostPath:
path: /var/lib/juicefs/volume
type: DirectoryOrCreate
name: jfs-dir
- hostPath:
path: /etc/updatedb.conf
type: FileOrCreate
name: updatedb
- hostPath:
path: /var/jfsCache
type: DirectoryOrCreate
name: cachedir-0
- name: kube-api-access-h8ljl
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: '2024-07-26T05:38:50Z'
status: 'True'
type: Initialized
- lastProbeTime: null
lastTransitionTime: '2024-07-26T05:39:10Z'
status: 'True'
type: Ready
- lastProbeTime: null
lastTransitionTime: '2024-07-26T05:39:10Z'
status: 'True'
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: '2024-07-26T05:38:50Z'
status: 'True'
type: PodScheduled
containerStatuses:
- containerID: >-
containerd://f785832f5a94d0d3837f0c4df75e0c681cb6ca3e1aa6fe3ce7ede3c0ca8b5bca
image: 'docker.io/juicedata/mount:ce-v1.2.0'
imageID: >-
docker.io/juicedata/mount@sha256:b28eb957b79068d16ebc26ec31f1da78086decbe8e450a46ff328ddcad853810
lastState: {}
name: jfs-mount
ready: true
restartCount: 0
started: true
state:
running:
startedAt: '2024-07-26T05:39:09Z'
hostIP: 192.168.239.102
phase: Running
podIP: 10.233.0.206
podIPs:
- ip: 10.233.0.206
qosClass: Guaranteed
startTime: '2024-07-26T05:38:50Z'

apiVersion: v1
kind: Pod
metadata:
annotations:
juicefs-721821b385f4e8fee4038c1b9f389b3f2c0e5a71c2f65f60b45534e: >-
/var/lib/kubelet/pods/43b6f5ef-de1e-4532-ab56-c012bffd1ca4/volumes/kubernetes.io~csi/pvc-28f70adb-eeb3-4917-a14f-c2b08f06d9e1/mount
juicefs-uniqueid: juicefs-sc
juicefs-uuid: 46d322df-f1cf-473f-b69e-6d67c799a5da
k8s.v1.cni.cncf.io/network-status: |-
[{
"name": "kube-ovn",
"interface": "eth0",
"ips": [
"10.233.0.207"
],
"mac": "ce:bf:a7:b8:e8:a0",
"default": true,
"dns": {},
"gateway": [
"10.233.0.1"
]
}]
ovn.kubernetes.io/allocated: 'true'
ovn.kubernetes.io/cidr: 10.233.0.0/16
ovn.kubernetes.io/gateway: 10.233.0.1
ovn.kubernetes.io/ip_address: 10.233.0.207
ovn.kubernetes.io/logical_router: ovn-cluster
ovn.kubernetes.io/logical_switch: ovn-default
ovn.kubernetes.io/mac_address: 'ce:bf:a7:b8:e8:a0'
ovn.kubernetes.io/pod_nic_type: veth-pair
ovn.kubernetes.io/routed: 'true'
finalizers:
- juicefs.com/finalizer
labels:
app.kubernetes.io/name: juicefs-mount
juicefs-hash: 522ce254977d78df0aad6a2e286ecc6c1b493151777f41e27a34cee883c6d42
volume-id: juicefs-sc
name: juicefs-paas-pre-1.novalocal-juicefs-sc-uzmaza
namespace: paas-controll-juceisfs
resourceVersion: '14130987'
spec:
containers:
- command:
- sh
- '-c'
- >-
/usr/local/bin/juicefs format --storage=s3
--bucket=http://192.168.239.24:8080/paaspre
--access-key=B0273VHX8FO2FGW47ON3 --trash-days=1
--secret-key=${secretkey} ${metaurl} paas-juicefs-default

      /bin/mount.juicefs ${metaurl} /jfs/juicefs-sc-uzmaza -o
      metrics=0.0.0.0:9567
  env:
    - name: JFS_FOREGROUND
      value: '1'
  envFrom:
    - secretRef:
        name: juicefs-paas-pre-1.novalocal-juicefs-sc-uzmaza-secret
  image: 'dockerhub.vertexwan.cn/pub/juicefsmountpod:ce-v1.2.0'
  imagePullPolicy: IfNotPresent
  lifecycle:
    preStop:
      exec:
        command:
          - sh
          - '-c'
          - +e
          - >-
            umount -l /jfs/juicefs-sc-uzmaza; rmdir /jfs/juicefs-sc-uzmaza;
            exit 0
  name: jfs-mount
  ports:
    - containerPort: 9567
      name: metrics
      protocol: TCP
  resources:
    limits:
      cpu: '1'
      memory: 2Gi
    requests:
      cpu: '1'
      memory: 2Gi
  securityContext:
    privileged: true
    runAsUser: 0
  terminationMessagePath: /dev/termination-log
  terminationMessagePolicy: File
  volumeMounts:
    - mountPath: /jfs
      mountPropagation: Bidirectional
      name: jfs-dir
    - mountPath: /etc/updatedb.conf
      name: updatedb
    - mountPath: /var/jfsCache
      name: cachedir-0
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-xrh8w
      readOnly: true

dnsPolicy: ClusterFirstWithHostNet
enableServiceLinks: true
hostname: pvc-28f70adb-eeb3-4917-a14f-c2b08f06d9e1
nodeName: paas-pre-1.novalocal
preemptionPolicy: PreemptLowerPriority
priority: 2000001000
priorityClassName: system-node-critical
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: juicefs-csi-node-sa
serviceAccountName: juicefs-csi-node-sa
terminationGracePeriodSeconds: 10
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/disk-pressure
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/memory-pressure
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/pid-pressure
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/unschedulable
operator: Exists
volumes:
- hostPath:
path: /var/lib/juicefs/volume
type: DirectoryOrCreate
name: jfs-dir
- hostPath:
path: /etc/updatedb.conf
type: FileOrCreate
name: updatedb
- hostPath:
path: /var/jfsCache
type: DirectoryOrCreate
name: cachedir-0
- name: kube-api-access-xrh8w
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: '2024-07-26T05:38:50Z'
status: 'True'
type: Initialized
- lastProbeTime: null
lastTransitionTime: '2024-07-26T05:39:11Z'
status: 'True'
type: Ready
- lastProbeTime: null
lastTransitionTime: '2024-07-26T05:39:11Z'
status: 'True'
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: '2024-07-26T05:38:50Z'
status: 'True'
type: PodScheduled
containerStatuses:
- containerID: >-
containerd://a04158bac8f3db4a7d49b45757fd829409eb8dd61db613298842559e52e675b7
image: 'docker.io/juicedata/mount:ce-v1.2.0'
imageID: >-
docker.io/juicedata/mount@sha256:b28eb957b79068d16ebc26ec31f1da78086decbe8e450a46ff328ddcad853810
lastState: {}
name: jfs-mount
ready: true
restartCount: 0
started: true
state:
running:
startedAt: '2024-07-26T05:39:10Z'
hostIP: 192.168.239.102
phase: Running
podIP: 10.233.0.207
podIPs:
- ip: 10.233.0.207
qosClass: Guaranteed
startTime: '2024-07-26T05:38:50Z'

@zwwhdls zwwhdls mentioned this issue Aug 1, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: update pod lock to pod hash juicedata/juicefs-csi-driver
2 participants
@luoxingit @zwwhdls