Specify Re-encrypt TLS Termination (i.e., Upstream TLS)

[candita]

Currently only Terminate and Passthrough TLS mode types are specified by the Gateway API. Without Upstream TLS, we cannot offer a comparable experience to our users. The absence of re-encrypt was once debated as a beta “blocker” in #968, and it warrants discussion to be included in the next release.

Upstream TLS, or re-encrypted, traffic is encrypted from the edge to gateway, followed by a re-encrypt before delivery from the gateway to the backend. While Gateway API specifies the edge->gateway hop, it doesn’t specify a way to encrypt the gateway->backend hop of the traffic path. It is missing a mechanism for separately providing the destination CA (certificate authority). It may be missing other components, as mentioned in #1067.

Adding upstreamTLS to the Listener tls.mode types, plus an additional, optional CACertificateRefs might be a good basic start. First it’s worth mentioning how re-encrypt is handled by other technologies. Istio, OpenShift, and Contour handle it somewhat differently as outlined below.

Istio Gateway appears to support re-encrypt with a DestinationRule to encrypt the gateway->backend hop:

• A secret representing a certificate/key pair, where the certificate is valid for the route host
• Set Gateway spec.servers[].port.protocol: HTTPS, spec.servers[].tls.mode=SIMPLE, spec.servers[].tls.credentialName:
• Set DestinationRule spec.trafficPolicy.tls.mode: SIMPLE
  Ref: Istio / Understanding TLS Configuration and Istio / Destination Rule

OpenShift Route (comparable to GW API Gateway) supports re-encrypt with the following route configuration items:

• A certificate/key pair, where the certificate is valid for the route host
• A separate destination CA certificate enables the Ingress Controller to trust the destination’s certificate
• An optional, separate CA certificate that completes the certificate chain
  Ref: Secured routes - Configuring Routes | Networking | OpenShift Container Platform 4.10

Contour supports re-encrypt from Envoy to the backend using:

• An Envoy client certificate
• A CA certificate and SubjectName which are both used to verify the backend endpoint’s identity.
• Kubernetes Service annotation: projectcontour.io/upstream-protocol.tls:
  Ref: Upstream TLS

Some of the ideas discussed so far for filling in this gap in Gateway API are documented in:

• TLS Termination Policy #52 issue for TLS Termination Policy
• tls: introduce mode and sni to cert matching behavior #256 implements one part of existing [SIG-NETWORK] TLS config in service-apis document section
• PR 256 overrode HTTPRoute: Add Reencrypt #81 but didn’t implement reencrypt
• Use of appProtocol discussion comment
• Use of ReferencePolicy at the Gateway level issue

[f5yacobucci]

NGINX Ingress Controller (NGINX/F5) supports re-encrypt via Policies attached to VirtualServer resources. The EgressMTLS fields allow for re-encrypt and mTLS client certificate validation. The CA and client certificate can be referenced as well as various other settings related to the TLS session. A user can specify if SNI should be used and can override which name is sent via sslName.

I am less familiar with the details, but community NGINX kubernetes ingress also supports re-encrypt use cases.

[howardjohn]

I am not sure how much this is just imprecise terminology vs actual design goals, but one thing I would note is "upstream TLS" is broader than "re-encrypt". Users may want to add TLS to traffic hops even if it wasn't TLS originally. Ex: client --- HTTP --> Gateway -- HTTPS --> server.

This is more common on egress where TLS is delegated to some centralized proxy

[knobunc]

Yes, the OpehShift terminology today is a little imprecise, but the reencrypt does support the use case you outlined (you can have an "insecure" external connection with a "secure" internal). As always, naming things is a battle :-)

[candita]

@howardjohn @knobunc Does it make sense to add a small definition, like:

A third termination mode, known as re-encrypt or Upstream TLS, allows the addition of TLS to traffic hops on the backend of the Gateway.

I prefer the term "re-encrypt" but I think "upstream TLS" is more common. I've also heard "downstream TLS" but don't know if that is equivalent to "re-encrypt".

[howardjohn]

I do not think upstream TLS belongs anywhere in Gateway. We are mixing downstream and upstream TLS here and they are entirely orthogonal. Upstream TLS is a property of the backend, while downstream TLS is a property of the frontend (Gateway)

[keithmattix]

+1, I'd much prefer this information to live in the result of #1282 or similar

[shaneutt]

I agree that we should support this case, it's an important one which has been discussed several times before, but we've yet to have a champion to drive it forward.

To add to the list of relevant issues we had the following issues and PRs where this was discussed also:

• Docs mentions Reencrypt for HTTPRoute and TLSRoute is available #968
• Clarifying TLS Reencryption Docs #1065

And the current documentation for this feature hints at the fact that this is not possible here (albeit the wording may be a bit confusing and may need some updating, it says we "can" do it then the next sentence says we can't).

Assuming this discussion continues to include comments in favor of this, I would suggest that we start a GEP as a result of this discussion and work towards the precise implementation details so that we can try and get this implemented sooner rather than later. @candita is it feasible for you to drive that forward, and is that a responsibility you want?

[candita]

@shaneutt I would be happy to work on a GEP to drive this forward.

[shaneutt]

Sounds good, let us maintainers know if there's anything you need in terms of support to get this going.

[candita]

I was hoping to get some more feedback on how various implementers handle reencrypt/upstreamTLS, and I was expecting to get some critique of my basic proposal. Would like to hear from @hbagdi and @youngnick and anyone else when they get a chance.

[youngnick]

Sorry to be slow, @candita. I really appreciate your roundup of the implementations, it's awesome to have everything in the one place. In terms of actual config, it seems to me like we need at least a CA cert for the backend service(s), maybe with the ability to specify an SNI as well. In addition, it seems like we might want to be able to supply the identity to the Gateway (that is, a client keypair)?

However, I'm not in favor of adding something to the Listener yet, for two reasons:

Firstly, I'm worried about adding things to Listener while we're still figuring out what we're going to do in GEP-1282 (#1282), because once people start using it, it's difficult to migrate them away. I know we need to solve this problem soon, but I don't want to rush into something that we end up stuck with either.

Secondly, the TLS details for connecting to a backend seem to me like they should be tied to the backend, rather than the Listener, that's why I'd like to address #1282. Related to this, do we see that there might be a need for CA certs on a per-backend basis? If so, that's a good argument for pushing that config closer to the backend, via something like we're talking about in #1282.

To be clear, I think that both this issue and GEP-1282 are priorities, and we should definitely address them in v0.6.0, which I think we're hoping to get out before October (in time for Kubecon NA).

[candita]

@youngnick I don't currently see any problem focusing on GEP-1282 so I can help there instead of creating a separate GEP. Perhaps one of the main use cases should be re-encrypt within #1282.