Flaky Test: Functional tests on Linux "disposing listeners failed"

[JunTaoLuo]

Seeing this as part of our partner acquisition linux builds but I suspect they are flaky on all platforms:

Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.HttpsTests.EmptyRequestLoggedAsInformation

Disposing listeners failed
Expected: True
Actual:   False
   at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.HttpsTests.<EmptyRequestLoggedAsInformation>d__0.MoveNext() in /datadrive/169923259d6dcfec/Universe/.r/KestrelHttpServer/test/Microsoft.AspNetCore.Server.Kestrel.FunctionalTests/HttpsTests.cs:line 56
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.HttpsTests.EmptyRequestLoggedAsInformation

Disposing listeners failed
Expected: True
Actual:   False
   at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.HttpsTests.<EmptyRequestLoggedAsInformation>d__0.MoveNext() in /datadrive/169923259d6dcfec/Universe/.r/KestrelHttpServer/test/Microsoft.AspNetCore.Server.Kestrel.FunctionalTests/HttpsTests.cs:line 56
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

More info: http://aspnetci/viewLog.html?buildId=197969&tab=buildResultsDiv&buildTypeId=CoreCLR_Linux

[JunTaoLuo]

cc @CesarBS

[mikeharder]

Another failure, also on Partner Acquisition \ Build: Linux

http://aspnetci/viewLog.html?buildId=199630&tab=buildResultsDiv&buildTypeId=CoreCLR_Linux

[davidfowl]

I know why this fails now. I debugged the test and have a root cause. The fix unfortunately is part of the work that @halter73 is doing right now to move connection management into kestrel core.

[davidfowl]

Disposing listeners seems like a slowness issue. I'm still looking into it. Upping the timeout seems to help.

[davidfowl]

This was from a run on my machine. Seems like the 2.5 second timeout might be too short. Something got slower.

Hosting starting
Hosting started
Connection id "0HL40LNU9EH94" started.
Connection id "0HL40LNU9EH94" received FIN.
Connection id "0HL40LNU9EH94" disconnecting.
Connection id "0HL40LNU9EH94" stopped.
Hosting shutdown
Disposing listener
Disposing 1 listeners
Disposing listener callback in 2223.7629ms
Walked 0 connections

Another run was fast

Disposing listener callback in 43.8965ms

Seems like libuv is taking a long time to schedule the work:

Hosting starting
Hosting started
Connection id "0HL40LTKQ3MGQ" started.
Connection id "0HL40LTKQ3MGQ" received FIN.
Connection id "0HL40LTKQ3MGQ" disconnecting.
Connection id "0HL40LTKQ3MGQ" stopped.
Hosting shutdown
Disposing listener
Disposing 1 listeners
Disposing listeners failed
Walked 0 connections
Work took 10234.49ms
Disposing listener callback in 10235.4492ms

[mikeharder]

Another failure: http://aspnetci/viewLog.html?buildTypeId=CoreCLR_Linux&buildId=206363

[davidfowl]

@halter73 What if we removed the Log "Dispose listeners failed"?

[davidfowl]

I increased the timeout but if it keeps happening we're going to have to tweak this test to avoid looking for this error.

[Eilon]

@davidfowl - anything left for preview1? Or would we juts fix things in dev at this point?

[davidfowl]

Going to close this until it repros again.

/cc @mikeharder if it repros open this back up.

[davidfowl]

Saw this again on my linux box while running it in a loop over 2 days..