add retry for detach azure disk

[andyzhangx]

What type of PR is this?
/kind bug

What this PR does / why we need it:
Current azure cloud provider would fail to detach azure disk when there is server side error, need to add retry mechanism for detach disk operation, while for attach disk operation, it's not necessary since k8s pv-controller will try if first try failed.

Which issue(s) this PR fixes:

Fixes #74396

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

fix issue: fail to detach azure disk when there is server side error

/kind bug
/assign @feiskyer
/priority important-soon
/sig azure

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andyzhangx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/cloudprovider/providers/azure/OWNERS [andyzhangx]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[feiskyer]

/lgtm

[andyzhangx]

BTW, if move 8 disks from one node to another in parellel, there will be always one disk not detched(only one), the error is like following:

I0222 08:38:22.503087       1 azure_controller_vmss.go:140] azureDisk - update(andy-vmss1124) backing off: vm(k8s-agentpool-24194600-vmss000001) detach disk(, /subscriptions/xxx/resourceGroups/andy-vmss1124/providers/Microsoft.Compute/disks/andy-vmss1124-dynamic-pvc-66815035-35ac-11e9-921c-000d3a002125), err: compute.VirtualMachineScaleSetVMsClient#Update: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="AttachDiskWhileBeingDetached" Message="Cannot attach data disk '763c0d05-4ae5-4699-8ce8-9c8ced5283da' to VM 'k8s-agentpool-24194600-vmss_1' because the disk is currently being detached or the last detach operation failed. Please wait until the disk is completely detached and then try again or delete/detach the disk explicitly again."

The detach action finally failed, it's more like attach a disk to node#1 would cause detach that disk from node#2 to fail which is not reasonable, I will contact with azure disk RP to check with this issue.

[feiskyer]

Then let's hold a while for root causes.

/hold

[feiskyer]

/test pull-kubernetes-e2e-aks-engine-azure

[andyzhangx]

the funny thing is Cannot attach data disk '763c0d05-4ae5-4699-8ce8-9c8ced5283da' to VM 'k8s-agentpool-24194600-vmss_1', while I never have data disk '763c0d05-4ae5-4699-8ce8-9c8ced5283da' in my resource group, and I could repro that easily in two vmss k8s clusters.

[andyzhangx]

@feiskyer The issue I could repro is related to VMSS, while I still insist on adding this retry logic only for detach disk (not necessary for attach disk since k8s controller will retry if failed) in case there is any potential issue, thus azure cloud provider could have more chance to retry, although in this case, it does not work perfectly. what's your opinion?

[feiskyer]

The issue I could repro is related to VMSS, while I still insist on adding this retry logic only for detach disk (not necessary for attach disk since k8s controller will retry if failed) in case there is any potential issue, thus azure cloud provider could have more chance to retry, although in this case, it does not work perfectly. what's your opinion?

Agreed. Let's wait a while for VMSS responses, in case there're still other potential issues.

[andyzhangx]

the failed error is most likely due to slow disk attach/detach on VMSS, let's merge this PR first since it would mitigate the issue a little
/hold cancel

[antoineco]

@andyzhangx @feiskyer I see there was no final conclusion on the issue, but in some clusters we can't seem to be able to get out of that "AttachDiskWhileBeingDetached" error loop without deleting affected Pods and giving Azure enough time to clean up the mess (detach disks from their VMSS instance).

[andyzhangx]

@antoineco the AttachDiskWhileBeingDetached issue on VMSS is already fixed in VMSS RP in mid April. This PR fixed the issue when disk detach operation failed and the disk is never detached on the original node.

[antoineco]

@andyzhangx thanks for the quick answer, unfortunately I'm not familiar with the "RP" acronym sorry (😅).

What I'm currently observing is a seemingly infinite loop of:

Code="AttachDiskWhileBeingDetached"
Message="Cannot attach data disk 'foo-bar-123' to VM 'mycluster-vmss_0' because the disk is currently being detached or the last detach operation failed. Please wait until the disk is completely detached and then try again or delete/detach the disk explicitly again."

I've tried waiting for over an hour sometimes, but for some reason the retry logic is not enough. Detaching disks manually (e.g. via the az CLI) returns the same error.

[andyzhangx]

@antoineco can you open an issue and also provide details info:

1. k8s version
2. region

And what's the status of disk foo-bar-123? is it attached or unattached?
if that disk is attached, you could also provide info by az vmss show -g <RESOURCE_GROUP_NAME> --name <VMSS_NAME> --instance-id <ID(number)> to show which VM is that disk attached to.

[andyzhangx]

@antoineco you may also file an azure support ticket for this issue, and paste this github link, and azure team would analyze your issue first, pls also provide my suggested info, thanks.

[antoineco]

@andyzhangx I will open an issue, just let me drop that here first for future reference:

• Kubernetes version: it's complicated (I'll explain, but we're testing 1.12 too)
• Regions: eastus, westeurope

"diskState": "Unattached"
(sometimes the disk is attached and impossible to detach for > 20 min)

[andyzhangx]

@andyzhangx I will open an issue, just let me drop that here first for future reference:

• Kubernetes version: it's complicated (I'll explain, but we're testing 1.12 too)
• Regions: eastus, westeurope

"diskState": "Unattached"
(sometimes the disk is attached and impossible to detach for > 20 min)

pls also cherry pick this PR: #74398, it fixed this issue: (sometimes the disk is attached and impossible to detach for > 20 min), only need to patch the kube-controller-manager

BTW, recently we found slow disk attach/detach issue when disk num is large, how many disks are attached to one VM?

[antoineco]

It's been cherry-picked but I documented it as #74581 (all references are to 1.12 cherry-picks). Maybe I lost something important in the conflict resolution and need to review again.

Each node usually has about 5 disks attached, which should be fairly reasonable.

[antoineco]

Ref. #78172