Implement basic AWS CloudHSM support for root CA creation + rewrite "FulcioCA" to "PKCS11CA" #187

mbestavros · 2021-09-17T04:08:27Z

This adds basic support for self-provisioning a root CA with AWS CloudHSM through the existing PKCS11 support. Since the "FulcioCA" option now encompasses both BYOD HSMs and AWS CloudHSM, it's been renamed to the more all-encompassing "PKCS11CA".

Signed-off-by: Mark Bestavros [email protected]

How it works

AWS CloudHSM provides a PKCS#11 compliant library for interacting with instances of its CloudHSM product. For the most part, this serves as a drop-in replacement for the SoftHSM PKCS#11 library; as long as keys are set up beforehand, just as Fulcio expects to be done when using SoftHSM, it (mostly) works with no code changes.

The exception is that CloudHSM does not support certificate storage on their HSMs:

AWS CloudHSM does not store certificates on the HSM, as certificates are public, non-confidential data.

In its current state, Fulcio does attempt to store the root CA it creates as part of the createca command within the HSM it's using, and this causes an error when running with AWS CloudHSM.

I've explored many possible workarounds for this limitation, and run into just as many dead ends. While those might change in the future, for now, this is the most simple working implementation of something that works with AWS: effectively, during root CA creation when running on AWS, instead of attempting to store the root CA in the HSM or in some other certificate storage service, it's just stored on-disk as a PEM file. Then, later on, when running fulcio serve, another option is exposed to load the root CA from disk instead of the HSM. This works around the lack of certificate support on AWS' HSMs.

Setup

AWS provides a rather comprehensive setup guide for getting a CloudHSM instance deployed and setting up an EC2 instance that can interact with it. For a complete setup, you'll want to make sure the CloudHSM Management Utility, Key Management Utility, and CloudHSM's PKCS#11 library are all installed on your EC2 instance.

First, you'll need to make sure your HSM is set up using the Management Utility using this guide. Next, you should provision keys within the HSM for Fulcio to use. Do this with the Key Management Utility:

Log in: loginHSM -u CU -s <username> -p <password>
Generate a keypair with appropriate label and ID: genECCKeyPair -i 14 -l PKCS11CA -id 1

With these steps done, your HSM is set up!

Next: clone Fulcio with this patch included onto the EC2 instance that has the AWS PKCS11 library installed. You'll also need to adjust config/crypto11.conf as such, in order to make it work with the CloudHSM setup:

{
    "Path" : "/opt/cloudhsm/lib/libcloudhsm_pkcs11.so",
    "TokenLabel": "cavium",
    "Pin" : "<username>:<password>"
}

The Path variable replaces the softhsm PKCS11 library with AWS CloudHSM's. On AWS CloudHSM, the TokenLabel must always be cavium - or things won't work. The Pin takes the format of a crypto user and corresponding password on the HSM, as seen here.

With this done, you're ready to use Fulcio! Here are some example commands:

fulcio createca --org=acme --country=US --locality=SomeTown --province=SomeProvince --postal-code=XXXX --street-address=XXXX --hsm-caroot-id 1 --out myrootCA.pem --hsm=aws`

This command creates a new root CA using the private key stored in AWS CloudHSM and stores it in the myrootCA.pem file locally.

fulcio serve --ca pkcs11ca --hsm-caroot-id 99 --aws_hsm_root_ca_path myrootCA.pem

And this command uses the generated myrootCA.pem file to run the Fulcio server.

lukehinds · 2021-09-17T09:05:59Z

wow, now that is how to write up a PR!

This looks good to me Mark, only thing I would add is to make additions to the README (such you have aboe for using libcloudhsm_pkcs11), I understand this may be pending though as its an WIP PR.

@dlorenc / @bobcallaway on ideas on the root CA being stored locally, I see no is security impact as the priv key is in the HSM. This is not to different to what we have with the GCP public good instance. I guess the main aspect is if someone wants this to be highly available. I am ok with addressing that later when we get user feedback to suggest adoption, we can then look to use an ec2 bucket api.

cmd/app/createca.go

pkg/api/pkcs11ca_signing_cert.go

bobcallaway · 2021-09-17T13:27:02Z

cmd/app/root.go

@@ -46,7 +46,8 @@ func Execute() {

 func init() {
 	rootCmd.PersistentFlags().StringVar(&logType, "log_type", "dev", "logger type to use (dev/prod)")
-	rootCmd.PersistentFlags().String("ca", "", "googleca | fulcioca")
+	rootCmd.PersistentFlags().String("ca", "", "googleca | pkcs11ca")
+	rootCmd.PersistentFlags().String("aws_hsm_root_ca_path", "", "Path to root CA on disk (only used with AWS HSM)")
 	rootCmd.PersistentFlags().String("gcp_private_ca_parent", "", "private ca parent: /projects/<project>/locations/<location>/<name> (only used with --ca googleca)")
 	rootCmd.PersistentFlags().String("hsm-caroot-id", "", "HSM ID for Root CA (only used with --ca fulcio)")


Suggested change

rootCmd.PersistentFlags().String("hsm-caroot-id", "", "HSM ID for Root CA (only used with --ca fulcio)")

rootCmd.PersistentFlags().String("hsm-caroot-id", "", "HSM ID for Root CA (only used with --ca pkcs11ca)")

bobcallaway · 2021-09-17T13:29:02Z

pkg/api/pkcs11ca_signing_cert.go

+		}
+		block, _ := pem.Decode(pubPEMData)
+		if block == nil || block.Type != "CERTIFICATE" {
+			log.Logger.Fatal("failed to decode PEM block containing certificate")


the intent here is good, but would cause a dropped connection to the client. If we did this once at initialization, that address this issue as well

mbestavros · 2021-09-23T18:10:15Z

@lukehinds @bobcallaway I've just pushed a few changes that address most of the minor comments people have made. I've also amended my setup guide and put it in its own .md file for posterity.

On another note, I think that it would be helpful to reorganize/simplify the various pieces of documentation in a follow up PR. As is, they're kind of just sitting around.

bobcallaway

minor nits, and opened separate issue for fixing per-transaction loading. otherwise LGTM

bobcallaway · 2021-10-05T23:59:58Z

cmd/app/createca.go

+		if viper.GetString("hsm") == "aws" {
+			log.Logger.Info("Running in AWS mode; skipping root CA storage in HSM")
+			if !viper.IsSet("out") {
+				log.Logger.Info("WARNING: --out is not set. Root CA will not be saved.")


Should this log level be a bit higher? If you're putting WARNING in the message, should it at least be log.Logger.Warn?

bobcallaway · 2021-10-06T00:00:27Z

cmd/app/createca.go

+			err = p11Ctx.ImportCertificateWithLabel(rootID, []byte(LABEL), certParse)
+			if err != nil {
+				log.Logger.Fatal(err)
+			}


Suggested change

err = p11Ctx.ImportCertificateWithLabel(rootID, []byte(LABEL), certParse)

if err != nil {

log.Logger.Fatal(err)

}

if err = p11Ctx.ImportCertificateWithLabel(rootID, []byte(LABEL), certParse); err != nil {

log.Logger.Fatal(err)

}

bobcallaway · 2021-10-06T00:07:44Z

cmd/app/createca.go

@@ -149,6 +156,7 @@ func init() {
 	createcaCmd.PersistentFlags().String("street-address", "", "Locality name for root CA")
 	createcaCmd.PersistentFlags().String("postal-code", "", "Locality name for root CA")
 	createcaCmd.PersistentFlags().String("out", "", "output root CA to file")
+	createcaCmd.PersistentFlags().String("hsm", "softhsm", "The HSM provider to use. Valid values: aws, softhsm (default)")


Suggested change

createcaCmd.PersistentFlags().String("hsm", "softhsm", "The HSM provider to use. Valid values: aws, softhsm (default)")

createcaCmd.PersistentFlags().String("hsm", "softhsm", "The HSM provider to use. Valid values: softhsm (default), aws")

This adds basic support for self-provisioning a root CA with AWS CloudHSM through the existing PKCS11 support. Since the "FulcioCA" option now encompasses both BYOD HSMs and AWS CloudHSM, it's been renamed to the more all-encompassing "PKCS11CA". Signed-off-by: Mark Bestavros <[email protected]>

mbestavros · 2021-10-13T05:51:11Z

Thanks @bobcallaway for the feedback - @lukehinds ready to merge?

lukehinds

lgtm, will merge this week just in case @bobcallaway has any further notes

bobcallaway

lgtm

mbestavros changed the title ~~Rewrite "FulcioCA" to "PKCS11CA" and add AWS support~~ [WIP] Implement basic AWS CloudHSM support for root CA creation + rewrite "FulcioCA" to "PKCS11CA" Sep 17, 2021

mbestavros marked this pull request as draft September 17, 2021 04:10

cpanato requested a review from lukehinds September 17, 2021 07:41

bobcallaway reviewed Sep 17, 2021

View reviewed changes

mbestavros force-pushed the local-disk branch from 5072e69 to df8793c Compare September 23, 2021 18:07

mbestavros marked this pull request as ready for review September 30, 2021 15:06

mbestavros changed the title ~~[WIP] Implement basic AWS CloudHSM support for root CA creation + rewrite "FulcioCA" to "PKCS11CA"~~ Implement basic AWS CloudHSM support for root CA creation + rewrite "FulcioCA" to "PKCS11CA" Sep 30, 2021

lukehinds mentioned this pull request Oct 5, 2021

Release 1.0 planning #191

Closed

4 tasks

bobcallaway mentioned this pull request Oct 6, 2021

load root CA from HSM at initialization rather than on per-transaction basis #196

Closed

bobcallaway reviewed Oct 6, 2021

View reviewed changes

mbestavros force-pushed the local-disk branch from df8793c to 46a0e09 Compare October 13, 2021 05:49

lukehinds approved these changes Oct 13, 2021

View reviewed changes

bobcallaway approved these changes Oct 13, 2021

View reviewed changes

bobcallaway merged commit fd319cf into sigstore:main Oct 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement basic AWS CloudHSM support for root CA creation + rewrite "FulcioCA" to "PKCS11CA" #187

Implement basic AWS CloudHSM support for root CA creation + rewrite "FulcioCA" to "PKCS11CA" #187

mbestavros commented Sep 17, 2021 •

edited

Loading

lukehinds commented Sep 17, 2021 •

edited

Loading

bobcallaway Sep 17, 2021

bobcallaway Sep 17, 2021

mbestavros commented Sep 23, 2021

bobcallaway left a comment

bobcallaway Oct 5, 2021

bobcallaway Oct 6, 2021

bobcallaway Oct 6, 2021

mbestavros commented Oct 13, 2021

lukehinds left a comment

bobcallaway left a comment

	rootCmd.PersistentFlags().String("hsm-caroot-id", "", "HSM ID for Root CA (only used with --ca fulcio)")
	rootCmd.PersistentFlags().String("hsm-caroot-id", "", "HSM ID for Root CA (only used with --ca pkcs11ca)")

	createcaCmd.PersistentFlags().String("hsm", "softhsm", "The HSM provider to use. Valid values: aws, softhsm (default)")
	createcaCmd.PersistentFlags().String("hsm", "softhsm", "The HSM provider to use. Valid values: softhsm (default), aws")

Implement basic AWS CloudHSM support for root CA creation + rewrite "FulcioCA" to "PKCS11CA" #187

Implement basic AWS CloudHSM support for root CA creation + rewrite "FulcioCA" to "PKCS11CA" #187

Conversation

mbestavros commented Sep 17, 2021 • edited Loading

How it works

Setup

lukehinds commented Sep 17, 2021 • edited Loading

bobcallaway Sep 17, 2021

Choose a reason for hiding this comment

bobcallaway Sep 17, 2021

Choose a reason for hiding this comment

mbestavros commented Sep 23, 2021

bobcallaway left a comment

Choose a reason for hiding this comment

bobcallaway Oct 5, 2021

Choose a reason for hiding this comment

bobcallaway Oct 6, 2021

Choose a reason for hiding this comment

bobcallaway Oct 6, 2021

Choose a reason for hiding this comment

mbestavros commented Oct 13, 2021

lukehinds left a comment

Choose a reason for hiding this comment

bobcallaway left a comment

Choose a reason for hiding this comment

mbestavros commented Sep 17, 2021 •

edited

Loading

lukehinds commented Sep 17, 2021 •

edited

Loading