Function App Slot swap is failing with 'http ping' failure

[spmanjunath]

I have several Azure pipelines that was using the below task to perform Slot swamp successfully, but these tasks are suddenly failing, this i noticed yesterday (7th Dec, 22). The swap was working till couple of days before that, i suspect there is some breaking change, or, a bug that is causing the issue. Please suggest solution.

  steps:
  - task: AzureAppServiceManage@0
    displayName: 'Swap Slot'
    inputs:
      azureSubscription: ${{ parameters.serviceConnectionName }}
      Action: 'Swap Slots'
      WebAppName: '$(FunctionAppName)'
      ResourceGroupName: '$(ResourceGroupName)'
      SourceSlot: '$(SlotName)'

Error:

2022-12-07T12:45:26.1272981Z ##[warning]Error: Failed to update deployment history. Conflict (CODE: 409)
2022-12-07T12:45:26.1288473Z ##[error]Error: Failed to swap App Service '-func' slots - 'Staging' and 'production'. Error: ExpectationFailed - Cannot swap site slots for site '-func' because the 'Staging' slot did not respond to http ping. (CODE: 417)
2022-12-07T12:45:26.1358404Z ##[section]Finishing: Swap Slot

Investigative information

Please provide the following:

• Timestamp: 2022-12-07T12:45:26.1272981Z
• Function App version: v3
• Deployment: https://****-func-staging.scm.azurewebsites.net/api/deployments/77431670417116365
• Region: northueurope

Thanks.

[spmanjunath]

Below is the error when Swap is tried from the Azure Portal

Failed to complete swap between slot 'Staging' and slot 'production'. Error: {"_body":"{"Code":"ExpectationFailed","Message":"Cannot swap site slots for site 'xxxxx-func' because the 'Staging' slot did not respond to http ping.","Target":null,"Details":[{"Message":"Cannot swap site slots for site 'xxxxx-func' because the 'Staging' slot did not respond to http ping."},{"Code":"ExpectationFailed"},{"ErrorEntity":{"ExtendedCode":"04057","MessageTemplate":"Cannot swap site slots for site '{0}' because the '{1}' slot did not respond to http ping.","Parameters":[],"InnerErrors":[],"Details":[],"Code":"ExpectationFailed","Message":"Cannot swap site slots for site 'xxxxx-func' because the 'Staging' slot did not respond to http ping."}}],"Innererror":null}","status":417,"ok":false,"statusText":"OK","headers":{"cache-control":["no-cache"],"content-length":["685"],"content-type":["application/json; charset=utf-8"],"date":["Thu"," 08 Dec 2022 06:39:52 GMT"],"expires":["-1"],"pragma":["no-cache"],"server":["Microsoft-IIS/10.0"],"strict-transport-security":["max-age=31536000; includeSubDomains"],"x-aspnet-version":["4.0.30319"],"x-ms-correlation-request-id":["53debf45-fca2-4dad-8a84-18fd91d13321"],"x-ms-ratelimit-remaining-subscription-reads":["11896"],"x-ms-request-id":["53debf45-fca2-4dad-8a84-18fd91d13321"],"x-ms-routing-request-id":["SOUTHINDIA:20221208T063953Z:53debf45-fca2-4dad-8a84-18fd91d13321"],"x-powered-by":["ASP.NET"]},"type":2,"url":"https://management.azure.com/subscriptions/XXXf/resourceGroups/xxxxx-rg/providers/Microsoft.Web/sites/xxxxx-func/slots/Staging/operationresults/80bdec7c-96cd-4577-978c-440490f8d227?api-version=2018-11-01"}

[RafiqSheikh]

We are having similar issue since 6th Dec 2022. Manual swap did not work either.

[spmanjunath]

@jviau - Any updates on this? The pipelines are broken and its a critical error. Any suggestions to resolve this would be of much help. Thanks.

[JosiahSiegel]

@spmanjunath I only resolved this error by dropping and re-creating the slot. In your case, it would the "Staging" slot. I specified -replace in Terraform

[j-l-carlson]

We are also experiencing this same issue

[spmanjunath]

@JosiahSiegel- Deleting the Staging slot did not resolve the issue.
@jviau - It appears that this issue is being faced by multiple teams, is there a workaround for this? Is this being looked at?

[ramya894]

@spmanjunath Please find the below SO link for guidelines, if you still face the issue please write back to us.

https://learn.microsoft.com/en-us/answers/questions/894692/slot-swag-failing-with-error-34slot-did-not-respon.html

[spmanjunath]

@ramya894 - i had look into the referenced thread before creating this thread. There are at least 3 pipelines that suddenly started failing from 7th Dec, and a day before that the same pipeline was working, and some of these pipelines are in place and working for almost a year now.

[RafiqSheikh]

@ramya894
I tried with the new appettings, it did not work.

[btull89]

My team is also impacted by this issue. Our pipelines have been running for over a year and now all of a sudden they are failing due to this issue. The Azure Portal shows the deployment succeeded and our code was swapped correctly; it appears the issue may be a communication issue between Azure Devops and the Azure Portal.

[j-l-carlson]

Several of our teams are reporting this issue and we can also confirm that the linked solution regarding WEBSITES_ENABLE_APP_CACHE does not solve the issue.

[s-pilo]

We have tried all proposed solutions and are also having this issue, though sporadically.

[alin-devnet]

This started happening on 8th Dec for us and thought it was linked with the App Service planned maintenance that was ongoing, but the issue is still persisting and it became a major issue. It looks like it happens for resources in certain subscriptions, it's working fine in others, all deployed to the same location (North Europe).

[s-pilo]

This started happening on 8th Dec for us and thought it was linked with the App Service planned maintenance that was ongoing, but the issue is still persisting and it became a major issue. It looks like it happens for resources in certain subscriptions, it's working fine in others, all deployed to the same location (North Europe).

Are all functions in a given subscription having this issue for you? Not so for us, some work fine, some do not, I've compared any differences there might be and am unable to find any consistency.

[j-l-carlson]

This started happening on 8th Dec for us and thought it was linked with the App Service planned maintenance that was ongoing, but the issue is still persisting and it became a major issue. It looks like it happens for resources in certain subscriptions, it's working fine in others, all deployed to the same location (North Europe).

Are all functions in a given subscription having this issue for you? Not so for us, some work fine, some do not, I've compared any differences there might be and am unable to find any consistency.

For us, not all function apps within a subscription are producing this issue, but if one is, it is producing the error consistently. We are mostly seeing the error in East US.

[alin-devnet]

This started happening on 8th Dec for us and thought it was linked with the App Service planned maintenance that was ongoing, but the issue is still persisting and it became a major issue. It looks like it happens for resources in certain subscriptions, it's working fine in others, all deployed to the same location (North Europe).

Are all functions in a given subscription having this issue for you? Not so for us, some work fine, some do not, I've compared any differences there might be and am unable to find any consistency.

Didn't try all of them (we have like 15+), but when we tried to deploy like 7 of them they all failed.

[spmanjunath]

We have 3 of our Function apps deployed in North Europe and all of them are failing, they are all Windows OS, couple of them deployed on V3 host and one is in V4 version.

[mligtenberg]

Same for us, all our Functions in East US are reporting this issue.

[LazyDave76]

Currently seeing this problem in NE region.

[fabiocav]

Folks, this issue has been identified and the team is actively deploying a patch to address the problem. We'll provide updates here, but this should be addressed in a number of regions, so a retry is recommended.

[LazyDave76]

Folks, this issue has been identified and the team is actively deploying a patch to address the problem. We'll provide updates here, but this should be addressed in a number of regions, so a retry is recommended.

In which regions is this fixed? Retried this morning and still happening in North Europe for us.

[spmanjunath]

@fabiocav Is the patch applied to North Europe region? In my tests the slot swap is failing.

[sujampana-fadv]

@fabiocav is the patch applied for East US2?

[kevink369]

When logged into Azure Portal you can see what regions have issues yet:
https://app.azure.com/h/2M5V-TP0/813bad

The problem is we consumers don't know when the impacted regions will be patched because of rolling it out in a "staged fashion". Hopefully soon!?

[Rujith]

Estimated schedule

[fabiocav]

Thank you @ranandLandmark .

[thiDucTran]

still not working for eastus at 22h00 UTC 19-Dec-2022

[es-alt]

Please provide schedule update. Still not working in US East as of Tuesday, Dec 12th, 2pm UTC.

[Jasonra102]

I'm still having problems with isolated instances in East US.

[j-l-carlson]

Yes, still not resolved for us in East US

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[kasper-bestseller]

I'm still seeing this issue:

Linux Consumption Plan EP1
Python Functions

I think this problem should be reopened.

[Jasonra102]

Same, seeing on isolated instances in East.

[robfrancis]

I came across this thread when I was experiencing a similar issue in a Devops Release pipeline.

ERROR: Cannot swap site slots for site '<NAME>' because the 'staging' slot did not respond to http ping.

Turned out the build pipeline used to create the release had the following in it,

steps:
  - task: DockerInstaller@0
    displayName: 'Install Docker 19.03.15'
    inputs:
      dockerVersion: 19.03.15

Despite specifying the version (and creating a build with no errors), the logs showed,

Starting: Install Docker 19.03.15
==============================================================================
Task         : Docker CLI installer
Description  : Install Docker CLI on agent machine.
Version      : 0.214.0
Author       : Microsoft Corporation
Help         : https://aka.ms/azpipes-docker-installer-tsg
==============================================================================
Downloading: https://download.docker.com/linux/static/stable/x86_64/docker-19.03.15.tgz
Extracting archive
/usr/bin/tar xC /home/vsts/work/_temp/66fa044c-3f47-4770-b4fc-afb1bec2dabc -f /home/vsts/work/_temp/docker-a62fff6e-7a0a-41de-83a0-872be10228eb.tgz
Caching tool: docker-stable 19.3.15 x64
Prepending PATH environment variable with directory: /opt/hostedtoolcache/docker-stable/19.3.15/x64
Verifying docker installation...
/usr/bin/docker --version
Docker version 20.10.22+azure-1, build 3a2c30b63ab20acfcc3f3550ea756a0561655a77
Finishing: Install Docker 19.03.15

Notice how the actual version installed is 20.10.22+azure-1 .

After updating the Docker pipeline step to - task: [email protected]

steps:
  - task: [email protected]
    displayName: 'Install Docker 19.03.15'
    inputs:
      dockerVersion: 19.03.15

Both the Build and Release pipelines worked correctly and the error went away.
microsoft/azure-pipelines-tasks#17594

I hope this helps someone.

[slouahchi]

With Azure DevOps Build Pipeline, we have the same issue since [email protected].
At the Build Pipeline, based on YAML, it is possible to Rollback to working [email protected].
But at the Azure DevOps Release Pipeline, based on the Visual Components, the DockerInstaller Task is fixed with Wildcard '0.*' to the latest version, currently [email protected].
Our Workaround at the Azure DevOps Release Pipelines is "continue with error" under "Control Section" and an additional Bash Task that sets installed Docker CLI to the PATH Variable.

Bash Task with Inline Code for our Hosted Agent for Docker 20.10.22 (x64)
`
export PATH=$PATH:/azp/_work/_tool/docker-stable/20.10.22/x64

chmod 777 /azp/_work/_tool/docker-stable/20.10.22/x64/docker
`

Please create an Hotfix for DockerInstaller with an higher Version than 0.214.0, or add Version 0.209.0 to the "Task Version" Field at the Azure DevOps Release Task. Alternative change "Task Version" Field from SelectBox to Input Field so that the default value "0.*" can be overridden by the user.