Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Certificate errors on .NET5/6 on Debian 11 (bullseye-slim) #62810

Closed
kevbite opened this issue Dec 14, 2021 · 17 comments
Closed

Certificate errors on .NET5/6 on Debian 11 (bullseye-slim) #62810

kevbite opened this issue Dec 14, 2021 · 17 comments
Labels
area-System.Security untriaged New issue has not been triaged by the area owner

Comments

@kevbite
Copy link

kevbite commented Dec 14, 2021

Description

After changing our base image for our aspnet core sites to use the newer Debian image (buster-slim to bullseye-slim) we seem to have started getting AuthenticationException when talking to external APIs that are using a certificate generate by a custom CA:

System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: NotSignatureValid, UntrustedRoot

System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: NotSignatureValid, UntrustedRoot
   at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---

We've not changed how we're copying our CA certificate.

COPY our-private-ca.crt /usr/local/share/ca-certificates/our-private-ca.crt
RUN chmod 644 /usr/local/share/ca-certificates/our-private-ca.crt
RUN update-ca-certificates --verbose

Reproduction Steps

  • Self Sign a certificate
  • Copy the CA for that certificate to a Debian 11 image
  • Call API with .NET code
  • View Exception

Expected behavior

Call to API works with self signed certificates when CA is copied to container image

Actual behavior

Exception is thrown

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

@dotnet-issue-labeler dotnet-issue-labeler bot added area-System.Security untriaged New issue has not been triaged by the area owner labels Dec 14, 2021
@ghost
Copy link

ghost commented Dec 14, 2021

Tagging subscribers to this area: @dotnet/area-system-security, @vcsjones, @krwq
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

After changing our base image for our aspnet core sites to use the newer Debian image (buster-slim to bullseye-slim) we seem to have started getting AuthenticationException when talking to external APIs that are using a self-signed certificate:

System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: NotSignatureValid, UntrustedRoot

We've not changed how we're copying our CA certificate.

COPY our-private-ca.crt /usr/local/share/ca-certificates/our-private-ca.crt
RUN chmod 644 /usr/local/share/ca-certificates/our-private-ca.crt
RUN update-ca-certificates --verbose

Reproduction Steps

  • Self Sign a certificate
  • Copy the CA for that certificate to a Debian 11 image
  • Call API with .NET code
  • View Exception

Expected behavior

Call to API works with self signed certificates when CA is copied to container image

Actual behavior

Exception is thrown

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

Author: kevbite
Assignees: -
Labels:

area-System.Security, untriaged

Milestone: -

@janvorli
Copy link
Member

I think it might be the known issue with Debian 11 running in a docker on a host with outdated libseccomp. See dotnet/dotnet-docker#3253. To verify that, can you try to open a shell in the debian 11 image (either the aspnet image or just vanilla debian 11) and attempt to run e.g. sudo apt-get update? If it fails it is the same issue.

@kevbite
Copy link
Author

kevbite commented Dec 30, 2021

Hey @janvorli, we run apt-get update for all the images that we create and we don't get any errors from them.
image

@janvorli
Copy link
Member

janvorli commented Jan 3, 2022

@kevbite thank you for your response. In that case, it is a different issue.
I've now noticed that the exception you are getting complains about The remote certificate is invalid because of errors in the certificate chain: NotSignatureValid, UntrustedRoot. So I wonder if it could be due to a root certificate that you depend on not being part of the docker image.

@bartonjs
Copy link
Member

bartonjs commented Jan 7, 2022

If you do something like

foreach (X509Certificate2 cert in (new X509Store(StoreName.Root, StoreLocation.LocalMachine, OpenFlags.ReadOnly)).Certificates)
{
    Console.WriteLine($"{cert.NotBefore}-{cert.NotAfter}: {cert.Subject}");
}

do you see your certificate? If not, then presumably bullseye changed something about the update-ca-certificates process.

If yes, then you can try something like

X509Certificate2 cert = new X509Certificate2("/usr/local/share/ca-certificates/our-private-ca.crt");
X509Chain chain = new X509Chain();
chain.ChainPolicy.RevocationMode = X509RevocationMode.NoCheck;

Console.WriteLine($"Chain builds successfully: {chain.Build(cert)}");

Console.WriteLine("Chain Element Status:");
foreach (X509ChainElement element in chain.ChainElements)
{
    Console.Write("  ");
    Console.WriteLine(element.Certificate.Subject);

    foreach (X509ChainStatus status in element.ChainElementStatus)
    {
        Console.WriteLine("    {0} ({1})", status.Status, status.StatusInformation);
    }

    Console.WriteLine();
}

Console.WriteLine("Chain Summary:");

foreach (X509ChainStatus status in chain.ChainStatus)
{
    Console.WriteLine("    {0} ({1})", status.Status, status.StatusInformation);
}

to see if there's anything interesting reported about it.

@kevbite
Copy link
Author

kevbite commented Jan 8, 2022

Hey @bartonjs

I've run the following tests by adding the code you supplied to the startup and these are the results comparing the base docker image and also the runtimes.

mcr.microsoft.com/dotnet/aspnet:5.0-buster-slim

08/12/2020 20:14:45-06/02/2023 20:14:45: C=GB, S=Scotland, L=Glasgow, OU=Platform Infrastructure, O=Company Ltd., [email protected], CN=company.internal
Chain builds successfully: True
Chain Element Status:
  C=GB, S=Scotland, L=Glasgow, OU=Platform Infrastructure, O=Company Ltd., [email protected], CN=company.internal
Chain Summary:

FROM mcr.microsoft.com/dotnet/aspnet:5.0-bullseye-slim

08/12/2020 20:14:45-06/02/2023 20:14:45: C=GB, S=Scotland, L=Glasgow, OU=Platform Infrastructure, O=Company Ltd., [email protected], CN=company.internal
Chain builds successfully: True
Chain Element Status:
  C=GB, S=Scotland, L=Glasgow, OU=Platform Infrastructure, O=Company Ltd., [email protected], CN=company.internal

Chain Summary:

FROM mcr.microsoft.com/dotnet/aspnet:6.0-bullseye-slim

08/12/2020 20:14:45-06/02/2023 20:14:45: C=GB, S=Scotland, L=Glasgow, OU=Platform Infrastructure, O=Company Ltd., [email protected], CN=company.internal
Chain builds successfully: True
Chain Element Status:
  C=GB, S=Scotland, L=Glasgow, OU=Platform Infrastructure, O=Company Ltd., [email protected], CN=company.internal

Chain Summary:

They all seem to find the certificate but they all also don't output anything for the Chain Summary.

@bartonjs
Copy link
Member

They all seem to find the certificate but they all also don't output anything for the Chain Summary.

Yeah, the Summary only contains things that went wrong. Since it counted as a trusted certificate on all of the configurations (and it's not expired, or more rare problems) the chain built successfully, and there were no errors to report.

So the problem isn't that your private root isn't getting trusted.

errors in the certificate chain: NotSignatureValid, UntrustedRoot

NotSignatureValid (which should really have been called SignatureNotValid, but whatever) means that the best candidate it could find at some level didn't have a public key that worked with the issued certificate's signature. Your best bet is probably to register a custom certificate verification routine, and print out the chain information (same as before, but starting with Console.WriteLine("Chain Element Status:");, since you don't want to call chain.Build again).

It may also be interesting to print what's in chain.ChainPolicy.ExtraStore, and the value of chain.ChainPolicy.DisableCertificateDownloads.

@kevbite
Copy link
Author

kevbite commented Jan 11, 2022

Hey @bartonjs, thanks for the help so far, I've just run the above tests, and this time we get completely different outputs between buster-slim and bullseye-slim. However, I'm not sure where we go from here. Below are all the outputs.

mcr.microsoft.com/dotnet/aspnet:5.0-buster-slim

Chain builds successfully: True
Chain Element Status:
  C=GB, S=Scotland, L=Glasgow, OU=Platform Infrastructure, O=Company Ltd., [email protected], CN=company.internal

  C=GB, S=Scotland, L=Glasgow, OU=Platform Infrastructure, O=Company Ltd., [email protected], CN=company.internal

Chain Summary:

chain.ChainPolicy.ExtraStore
chain.ChainPolicy.DisableCertificateDownloads = False

mcr.microsoft.com/dotnet/aspnet:5.0-bullseye-slim

Chain builds successfully: False
Chain Element Status:
  C=GB, S=Scotland, L=Glasgow, OU=Platform Infrastructure, O=Company Ltd., [email protected], CN=company.internal
    NotSignatureValid (certificate signature failure)
    UntrustedRoot (self signed certificate)

Chain Summary:
    NotSignatureValid (certificate signature failure)
    UntrustedRoot (self signed certificate)

chain.ChainPolicy.ExtraStore
chain.ChainPolicy.DisableCertificateDownloads = False

mcr.microsoft.com/dotnet/aspnet:6.0-bullseye-slim

Chain builds successfully: False
Chain Element Status:
  C=GB, S=Scotland, L=Glasgow, OU=Platform Infrastructure, O=Company Ltd., [email protected], CN=company.internal
    NotSignatureValid (certificate signature failure)
    UntrustedRoot (self signed certificate)

Chain Summary:
    NotSignatureValid (certificate signature failure)
    UntrustedRoot (self signed certificate)

chain.ChainPolicy.ExtraStore
chain.ChainPolicy.DisableCertificateDownloads = False

@vcsjones
Copy link
Member

@kevbite are you able to supply the public certificate here?

Another thing to try for your self-signed certificate in the container:

openssl verify -CAfile yourcert.crt -check_ss_sig yourcert.crt

This might print our additional details for what OpenSSL itself might not like the self-signed certificate. Perhaps we're running in to something like #47215.

@bartonjs
Copy link
Member

The mcr.microsoft.com/dotnet/aspnet:5.0-buster-slim output looks like your CA and your leaf cert have the exact same subject name.

My guess is that between buster and bullseye they upgraded versions of OpenSSL and it's no longer willing to build that chain.

@vcsjones
Copy link
Member

looks like your CA and your leaf cert have the exact same subject name.

I guess that is something I misunderstood, I assumed from the original post this is a self signed certificate.

@kevbite
Copy link
Author

kevbite commented Jan 11, 2022

@bartonjs yep you're correct they do both have the same subject 🤔

@vcsjones
Copy link
Member

vcsjones commented Jan 11, 2022

Do the certificates have an Authority Key Identifier / Subject Key Identifier extension? If not then this sounds like #31569.

@kevbite
Copy link
Author

kevbite commented Jan 11, 2022

We don't have either an Authority Key Identifier or Subject Key Identifier extension. Just reading through that post and it seems very similar, I'll investigate maybe getting these changed and try again. Thanks!

@kevbite
Copy link
Author

kevbite commented Jan 12, 2022

@vcsjones I think I missed one of your earlier messages about getting OpenSSL to verify the chain and this is the output between the 2 debain images.

mcr.microsoft.com/dotnet/aspnet:5.0-buster-slim

openssl version
OpenSSL 1.1.1d  10 Sep 2019

openssl verify -CAfile /usr/local/share/ca-certificates/private-ca.crt -check_ss_sig internal.crt
internal.crt: OK

mcr.microsoft.com/dotnet/aspnet:6.0-bullseye-slim

openssl version
OpenSSL 1.1.1k  25 Mar 2021

openssl verify -CAfile /usr/local/share/ca-certificates/private-ca.crt -check_ss_sig internal.crt
CN = company.internal, emailAddress = [email protected], O = Company Ltd., OU = Platform Infrastructure, L = Glasgow, ST = Scotland, C = GB
error 18 at 0 depth lookup: self signed certificate
CN = company.internal, emailAddress = [email protected], O = Company Ltd., OU = Platform Infrastructure, L = Glasgow, ST = Scotland, C = GB
error 7 at 0 depth lookup: certificate signature failure
error internal.crt: verification failed
140030207251776:error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:66:
140030207251776:error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:588:
140030207251776:error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:../crypto/asn1/a_verify.c:170:

@vcsjones
Copy link
Member

Since it reproduces in the OpenSSL CLI that explains the observed behavior change for .NET - we're just reporting what OpenSSL tells us. I think at this point the best course of action would be to

  1. Use different subjects for your CA and leaf certificates.
  2. Start including SKI and AKI extensions in your certificates.

Doing one or the other will probably resolve the issue, doing both is also fine.

@kevbite
Copy link
Author

kevbite commented Jan 14, 2022

Thanks for all the help, it's been great getting to the end of it. I've actually learned a lot along the way!

@ghost ghost locked as resolved and limited conversation to collaborators Feb 13, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-System.Security untriaged New issue has not been triaged by the area owner
Projects
None yet
Development

No branches or pull requests

4 participants