[BUG] delete volume job is failed

[huangguoqiang]

Uploading juicefs-csi-controller-0_juicefs-plugin.log…
What happened:
delete volume job is failed

What you expected to happen:
quickly delete volume

Anything else we need to know?
this log :I0531 17:16:30.687477 7 pod_mount.go:449] waitUtilJobCompleted: Job juicefs-4370bc01-delvol is completed
I0531 17:16:34.692467 7 pod_mount.go:449] waitUtilJobCompleted: Job juicefs-17089f91-delvol is completed
E0531 17:16:42.688278 7 provisioner.go:218] provisioner: delete vol error waitUtilJobCompleted: job juicefs-26aa4266-delvol isn't completed: 2024/05/31 17:16:03.714372 juicefs[7] : Volume is formatted as {
E0531 17:16:42.688306 7 controller.go:1482] delete "pvc-5c8882a3-847b-40e4-880d-ef2913579a54": volume deletion failed: unable to provision delete volume: waitUtilJobCompleted: job juicefs-26aa4266-delvol isn't completed: 2024/05/31 17:16:03.714372 juicefs[7] : Volume is formatted as {
E0531 17:16:42.688375 7 controller.go:1031] error syncing volume "pvc-5c8882a3-847b-40e4-880d-ef2913579a54": unable to provision delete volume: waitUtilJobCompleted: job juicefs-26aa4266-delvol isn't completed: 2024/05/31 17:16:03.714372 juicefs[7] : Volume is formatted as {
I0531 17:16:42.688387 7 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolume", Namespace:"", Name:"pvc-5c8882a3-847b-40e4-880d-ef2913579a54", UID:"5a28a7a6-af1a-438d-9738-cad80daafd62", APIVersion:"v1", ResourceVersion:"370197729", FieldPath:""}): type: 'Warning' reason: 'VolumeFailedDelete' unable to provision delete volume: waitUtilJobCompleted: job juicefs-26aa4266-delvol isn't completed: 2024/05/31 17:16:03.714372 juicefs[7] : Volume is formatted as {
E0531 17:16:57.087501 7 provisioner.go:218] provisioner: delete vol error waitUtilJobCompleted: job juicefs-fc0876fe-delvol isn't completed: 2024/05/31 17:16:18.103205 juicefs[7] : Volume is formatted as {
E0531 17:16:57.087523 7 controller.go:1482] delete "pvc-e36fcd01-b570-4617-b4ba-0bac10360324": volume deletion failed: unable to provision delete volume: waitUtilJobCompleted: job juicefs-fc0876fe-delvol isn't completed: 2024/05/31 17:16:18.103205 juicefs[7] : Volume is formatted as {
E0531 17:16:57.087568 7 controller.go:1031] error syncing volume "pvc-e36fcd01-b570-4617-b4ba-0bac10360324": unable to provision delete volume: waitUtilJobCompleted: job juicefs-fc0876fe-delvol isn't completed: 2024/05/31 17:16:18.103205 juicefs[7] : Volume is formatted as {
I0531 17:16:57.087614 7 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolume", Namespace:"", Name:"pvc-e36fcd01-b570-4617-b4ba-0bac10360324", UID:"2f14a035-634a-4796-891a-37b81417d27e", APIVersion:"v1", ResourceVersion:"370190284", FieldPath:""}): type: 'Warning' reason: 'VolumeFailedDelete' unable to provision delete volume: waitUtilJobCompleted: job juicefs-fc0876fe-delvol isn't completed: 2024/05/31 17:16:18.103205 juicefs[7] : Volume is formatted as {
E0531 17:17:12.288128 7 provisioner.go:218] provisioner: delete vol error waitUtilJobCompleted: job juicefs-fec29e38-delvol isn't completed: 2024/05/31 17:16:33.074298 juicefs[7] : Volume is formatted as {
job juicefs-fec29e38-delvol isn't completed: 2024/05/31 17:16:33.074298 juicefs[7] : Volume is formatted as {
E0531 17:17:12.288184 7 controller.go:1031] error syncing volume "pvc-8ce7e37d-9540-432e-9763-448c0dd88016": unable to provision delete volume: waitUtilJobCompleted: job juicefs-fec29e38-delvol isn't completed: 2024/05/31 17:16:33.074298 juicefs[7] : Volume is formatted as {
I0531 17:17:12.288416 7 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolume", Namespace:"", Name:"pvc-8ce7e37d-9540-432e-9763-448c0dd88016", UID:"49b36f09-97ca-4b24-b06d-4cb3cd2df1b3", APIVersion:"v1", ResourceVersion:"358856640", FieldPath:""}): type: 'Warning' reason: 'VolumeFailedDelete' unable to provision delete volume: waitUtilJobCompleted: job juicefs-fec29e38-delvol isn't completed: 2024/05/31 17:16:33.074298 juicefs[7] : Volume is formatted as {

and event show log:
26m Normal Killing pod/juicefs-rke-10-5-40-134-pvc-18179844-afee-407d-ad02-d26e6653e197-hxiesc Stopping container jfs-mount
26m Warning FailedPreStopHook pod/juicefs-rke-10-5-40-134-pvc-18179844-afee-407d-ad02-d26e6653e197-hxiesc Exec lifecycle hook ([sh -c umount /jfs/pvc-18179844-afee-407d-ad02-d26e6653e197-hxiesc -l && rmdir /jfs/pvc-18179844-afee-407d-ad02-d26e6653e197-hxiesc]) for Container "jfs-mount" in Pod "juicefs-rke-10-5-40-134-pvc-18179844-afee-407d-ad02-d26e6653e197-hxiesc_juicefs(b7736bc5-d5c4-46e9-9cf7-51239a8206e2)" failed - error: command 'sh -c umount /jfs/pvc-18179844-afee-407d-ad02-d26e6653e197-hxiesc -l && rmdir /jfs/pvc-18179844-afee-407d-ad02-d26e6653e197-hxiesc' exited with 137: , message: ""
60m Normal Killing pod/juicefs-rke-10-5-40-134-pvc-1e61ed83-87af-45a0-a8ae-6267cb4f7cc3-sloseu Stopping container jfs-mount
60m Warning FailedPreStopHook pod/juicefs-rke-10-5-40-134-pvc-1e61ed83-87af-45a0-a8ae-6267cb4f7cc3-sloseu Exec lifecycle hook ([sh -c umount /jfs/pvc-1e61ed83-87af-45a0-a8ae-6267cb4f7cc3-sloseu -l && rmdir /jfs/pvc-1e61ed83-87af-45a0-a8ae-6267cb4f7cc3-sloseu]) for Container "jfs-mount" in Pod "juicefs-rke-10-5-40-134-pvc-1e61ed83-87af-45a0-a8ae-6267cb4f7cc3-sloseu_juicefs(29b2375f-9736-45df-bcfb-77871141365a)" failed - error: command 'sh -c umount /jfs/pvc-1e61ed83-87af-45a0-a8ae-6267cb4f7cc3-sloseu -l && rmdir /jfs/pvc-1e61ed83-87af-45a0-a8ae-6267cb4f7cc3-sloseu' exited with 137: , message: ""
60m Normal Pulled pod/juicefs-rke-10-5-40-134-pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg Container image "juicedata/mount:ce-v1.1.1" already present on machine
60m Normal Created pod/juicefs-rke-10-5-40-134-pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg Created container jfs-mount
60m Normal Started pod/juicefs-rke-10-5-40-134-pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg Started container jfs-mount
59m Normal Killing pod/juicefs-rke-10-5-40-134-pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg Stopping container jfs-mount
59m Warning FailedPreStopHook pod/juicefs-rke-10-5-40-134-pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg Exec lifecycle hook ([sh -c umount /jfs/pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg -l && rmdir /jfs/pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg]) for Container "jfs-mount" in Pod "juicefs-rke-10-5-40-134-pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg_juicefs(e5d67dad-a222-4f1f-b894-f7f60368f41f)" failed - error: command 'sh -c umount /jfs/pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg -l && rmdir /jfs/pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg' exited with 137: , message: ""

Environment:

• JuiceFS CSI Driver version (which image tag did your CSI Driver use): 0.19.5
• Kubernetes version (e.g. kubectl version): 1.21.11
• Object storage (cloud provider and region): ceph s3
• Metadata engine info (version, cloud provider managed or self maintained): redis senstinel
• Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage):
• Others: