add an HTTP health check endpoint (mTLS exempted)

[dmitris]

Summary
Add the new /healthz HTTP-accessible entrypoint which would not use TLS certificates.

Description
The proposal below is copied from the Slack threat with slight edits. The proposed change is very similar to sigstore/fulcio#1258 "add HTTP and GRPC health check endpoints" - though for now without gRPC.

A question and suggestion regarding the TSA timestamp-server: Currently there are two entrypoints in its API : /api/v1/timestamp to post timestamp requests, and /api/v1/timestamp/certchain which returns the timestamping certificate chain. I suggest to add an additional simple “heartbeat” entrypoint that could be used for checking that the server is alive - /heartz (as in fulcio) that would return HTTP response 200 and a string OK if the server is running. A desirable and necessary property would be to be able to run the heartbeat responder on a separate port from the “main timestamping one” and be able exempt it from the TLS certificate or mTLS requirements.

I’m currently working to setup a cosign and TSA “cooperation” with the timestamp-server running in AWS, and plan to use mTLS for the cosign<-> timestamp-server connection. So I need the TSA to run using its TLS certificates for the web service, but also need to run an LB (NLB) that would do the health check. Currently I have to run timestamp-server like this:

timestamp-server serve --disable-ntp-monitoring --scheme https \
   --tls-ca /opt/dir/cacert.pem --tls-certificate /opt/dir/cert.pem --tls-key /opt/dir/key.pem \
   --tls-host 0.0.0.0 --tls-port 4443 --host 0.0.0.0 --port 80 

so that I could have cosign connecting to the TSA on port 4443 over mTLS but the LB health check using port 80 without mTLS. I also have to specify /api/v1/timestamp/certchain in the Health Check path since I don’t see anything else that could be used 😄 Sending the whole certificate chain to the health checks every 10 seconds seem unnecessary, and also due to the --host <host> --port 80 the whole timestamping server is available on port 80 without mTLS which is really against the security requirements, and also unnecessary. Therefore the proposal is to add a “dedicated” entry point (/healthz as in https://github.com/sigstore/fulcio/pull/1258/files#diff-7a709a99f5538c14e69f3d90368e405742552c40104fd88621ca2af751344356R70) that would be exempt from the TLS requirements and wouldn’t expose the "core" functionality. It could be done with additional command-line parameters: timestamp-server server […] --heartz-host 0.0.0.0 --heartz-port 80.

[malancas]

There is a /ping endpoint that returns a 200 status available, defined here. However it does run on the same port as the other endpoints.

[haydentherapper]

Summarizing what we chatted about offline:

• There is the existing /ping endpoint as @malancas pointed out. However this does not solve the use case of having the service served over TLS and a single healthcheck endpoint without TLS
• OpenAPI code generation is not currently configured to serve a subset of endpoints per host, so you have to serve the entire service on each listener, as you've noted
• One solution is to see if it's possible to configure which set of endpoints are supported per listener, so you could restrict everything but the healthcheck endpoint over non-TLS. You'll have to see if OpenAPI codegen supports this, or see if you can configure the service
• I am supportive of this change to have a dedicated host/port if you can't do the above, but I think you'll still run into openapi limitations when you go to declare the new handler.

[dmitris]

I think the objective of "allow cloud Load Balancers to do health check but the server must be available only over HTTPS" can be met with the following sample change. Obviously, it is "too drastic" - it completely precludes access to the timestamping functionality over HTTP, for someone like us it would be ok, but for others not. It should be possible to do such wrapper only if there is a command-line flags, like --http-ping-only, false by default - but I don't know yet how. Let me know what you think of this approach @haydentherapper.

[haydentherapper]

I'm supportive of this approach, this seems like a pretty clean solution. We should have a flag like what you proposed to make this opt-in, so that users can still run the service over HTTP if they want to.