-
Notifications
You must be signed in to change notification settings - Fork 4.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
.Net Core SocketsHttpHandler with Client Certificates hangs on windows #26708
Comments
@bartonjs I think we've seen problems before with using certificates from Azure key vault. Also, this code will produce an X509Certificate2 object that is missing the private key. I assume the Base64 here doesn't really have the private key. So, HttpClient won't be able to use it. private static X509Certificate2 GetCert()
{
var fabrikam = "MIIKNgIBAzCCCfY…";
var certBytes = Convert.FromBase64String(fabrikam);
return new X509Certificate2(certBytes, string.Empty, X509KeyStorageFlags.UserKeySet | X509KeyStorageFlags.EphemeralKeySet);
} |
@davidsh I ran that snippet of code (using the real base64 value) and |
Ok. Thanks. I didn't expect a repro to have a real private key with it. I'm assuming it is a test certificate of some kind. As far as why it doesn't work, I think this has something to do with @timmydo Can you use |
SslStream + EphemeralKeySet is #23749. |
Replacing the line with:
fixes it. Would be nice if the original didn't lock up though... Thanks for the help! |
Based on this discussion, this issue can be closed as a duplicate of #23749. |
@davidsh, that other issue appeared to manifest as exceptions. This one is manifesting as a hang, where the SendAsync doesn't complete, even after cancellation is requested and the connection is disposed. Do we understand why? Seems like there may be a secondary issue here. |
It's hanging due to a different bug (not fully root caused) in SocketsHttpHandler. Looking briefly in Visual Studio, I see an exception happening during TLS/SSL setup probably due to the use of the EmpheralKeySet:
Callstack:
Most likely this exception is being eaten and this is why the task is hanging (doesn't terminate with error). |
This looks like a bug in SslStream as well. The exception I show above is caused during a Re-negotiate of the TLS connection. Basically, the initial TLS connection works because client certificates are not requested right away. Then SocketsHttpHandler thinks the TLS connection is good and starts preparing to write request headers, etc. Afterwards the server decides to do the TLS re-negotiate and ask for client certificates. This is a common pattern that usually happens after the client makes an initial HTTP request specifying the exact path of the resource. It is at this point that the TLS stream breaks because the client certificate can't be read/sent to the server (due to the EmpheralKeySet). The first exception thrown "No credentials are available in the security package" is lost in SslStream and then the SslStream gets disposed. But all of this seems invisible to SocketsHttpHandler which isn't kept aware of the state of the connection. The first problem that needs to be solved is to properly expose the initial exception generated by the re-negotiate of the SslStream. Then the rest of the error handling needs to be fixed as well to propagate this info to the SocketsHttpHandler (and other callers). |
I am able to reproduce this hang with .NET Core 2.2. However, in .NET Core 3.0 current master (and Preview4), the hang does not occur. I get an exception:
Stacktrace>System.AggregateException: One or more errors occurred. (An error occurred while sending the request.) ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The read operation failed, see inner exception. ---> System.ComponentModel.Win32Exception: No credentials are available in the security package at System.Net.SSPIWrapper.AcquireCredentialsHandle(SSPIInterface secModule, String package, CredentialUse intent, SCHANNEL_CRED scc) at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(CredentialUse credUsage, SCHANNEL_CRED secureCredential) at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(X509Certificate certificate, SslProtocols protocols, EncryptionPolicy policy, Boolean isServer) at System.Net.Security.SecureChannel.AcquireClientCredentials(Byte[]& thumbPrint) at System.Net.Security.SecureChannel.GenerateToken(Byte[] input, Int32 offset, Int32 count, Byte[]& output) at System.Net.Security.SecureChannel.NextMessage(Byte[] incoming, Int32 offset, Int32 count) at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest) --- End of stack trace from previous location where exception was thrown --- at System.Net.Security.SslStream.ThrowIfExceptional() at System.Net.Security.SslStream.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck) at System.Net.Security.SslStream.CheckOldKeyDecryptedData(Memory`1 buffer) at System.Net.Security.SslStream.HandleQueuedCallback(Object& queuedStateRequest) --- End of stack trace from previous location where exception was thrown --- at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer) --- End of inner exception stack trace --- at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer) at System.Net.Http.HttpConnection.FillAsync() at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed) at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at ssltest.Program.DoIt() in s:\dotnet\ssltest\Program.cs:line 42 --- End of inner exception stack trace --- at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at System.Threading.Tasks.Task.Wait() at ssltest.Program.Main(String[] args) in s:\dotnet\ssltest\Program.cs:line 18 ---> (Inner Exception #0) System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The read operation failed, see inner exception. ---> System.ComponentModel.Win32Exception: No credentials are available in the security package at System.Net.SSPIWrapper.AcquireCredentialsHandle(SSPIInterface secModule, String package, CredentialUse intent, SCHANNEL_CRED scc) at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(CredentialUse credUsage, SCHANNEL_CRED secureCredential) at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(X509Certificate certificate, SslProtocols protocols, EncryptionPolicy policy, Boolean isServer) at System.Net.Security.SecureChannel.AcquireClientCredentials(Byte[]& thumbPrint) at System.Net.Security.SecureChannel.GenerateToken(Byte[] input, Int32 offset, Int32 count, Byte[]& output) at System.Net.Security.SecureChannel.NextMessage(Byte[] incoming, Int32 offset, Int32 count) at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest) --- End of stack trace from previous location where exception was thrown --- at System.Net.Security.SslStream.ThrowIfExceptional() at System.Net.Security.SslStream.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck) at System.Net.Security.SslStream.CheckOldKeyDecryptedData(Memory`1 buffer) at System.Net.Security.SslStream.HandleQueuedCallback(Object& queuedStateRequest) --- End of stack trace from previous location where exception was thrown --- at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer) --- End of inner exception stack trace --- at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer) at System.Net.Http.HttpConnection.FillAsync() at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed) at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at ssltest.Program.DoIt() in s:\dotnet\ssltest\Program.cs:line 42<---due to #23749. I think the refactoring work for SslStream with PR dotnet/corefx#35109 fixed the hang as a side-affect. |
We're using certs from azure key vault. I'm not sure what it does to the certs, but it seems to cause the httpclient to hang and not send the request. I created a self signed cert below and uploaded/downloaded from AKV to repro.
The text was updated successfully, but these errors were encountered: