TCP Loopback Fast Path causes occasional connection hangs

[halter73]

Edit by @CesarBS: original issue title was 'ZeroContentLengthSetAutomaticallyForNonKeepAliveRequests test failure'

I've only seen this once so far.

xUnit.net .NET CLI test runner (64-bit Desktop .NET win10-x64)
  Discovering: Microsoft.AspNetCore.Server.KestrelTests
  Discovered:  Microsoft.AspNetCore.Server.KestrelTests
  Starting:    Microsoft.AspNetCore.Server.KestrelTests
    Microsoft.AspNetCore.Server.KestrelTests.EngineTests.ZeroContentLengthSetAutomaticallyForNonKeepAliveRequests(testContext: TestServiceContext { App = null, AppLifetime = LifetimeNotImplemented { ApplicationStarted = (throws NotImplementedException), ApplicationStopped = (throws NotImplementedException), ApplicationStopping = (throws NotImplementedException) }, DateHeaderValue = "Tue, 04 Oct 2016 02:39:11 GMT", DateHeaderValueManager = DateHeaderValueManager { }, FrameFactory = null, ... }) [FAIL]
      System.IO.IOException : Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
      ---- System.Net.Sockets.SocketException : An existing connection was forcibly closed by the remote host
      Stack Trace:
           at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
           at System.Threading.Tasks.TaskFactory`1.FromAsyncTrimPromise`1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at System.IO.StreamReader.<ReadBufferAsync>d__97.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at System.IO.StreamReader.<ReadAsyncInternal>d__64.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
        C:\Users\shalter\dev\Universe\KestrelHttpServer\test\shared\TestConnection.cs(118,0): at Microsoft.AspNetCore.Testing.TestConnection.<ReceiveEnd>d__11.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
        EngineTests.cs(497,0): at Microsoft.AspNetCore.Server.KestrelTests.EngineTests.<ZeroContentLengthSetAutomaticallyForNonKeepAliveRequests>d__18.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        ----- Inner Stack Trace -----
           at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
           at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
  Finished:    Microsoft.AspNetCore.Server.KestrelTests
=== TEST EXECUTION SUMMARY ===
   Microsoft.AspNetCore.Server.KestrelTests  Total: 1056, Errors: 0, Failed: 1, Skipped: 0, Time: 144.337s

SUMMARY: Total: 2 targets, Passed: 1, Failed: 1.

[cesarblum]

I was able to repro this and it's pretty weird.

In order to repro I put a while (true) around the using block that creates the test connection. I also enabled logging to see what was going on.

What was happening was that very few connections were being created, and then the test hung on the call to ReadAsync() in TestConnection.ReceiveEnd() i.e. it hung waiting for the FIN from the server. The server did send the FIN in those cases though. After ~2 minutes the test fails with the exception reported above.

There are two ways to make the issue go away:

1. Use TestConnection.SendEnd() in the test instead of Send(). Alternatively, call _socket.Shutdown(SocketShutdown.Send) in TestConnection.ReceiveEnd() before the call to ReadAsync().

2. Remove the SIO_LOOPBACK_FAST_PATH stuff from TestConnection.CreateConnectedLoopbackSocket.

So it's clearly SIO_LOOPBACK_FAST_PATH doing something weird around connection termination.

I'll send a PR in a few moments.