Setting up BIG-IP with AWS CloudHSM

Recently I was working on a project and there was a requirement for using AWS CloudHSM.  F5 has documented the process to install the AWS CloudHSM client in the implementation guide. I found it light on details of working config and lacked an example.  So let's pickup where the article leaves you on having installed the client software and answer the question. What does a working configuration look like? 

AWS Documentation SDK 3

F5 has validated the 3.2 client for integration with BIG-IP.  In our implementation guide we say install the client and for configuration of said software go to AWS documentation. If you are using the AWS website you may be scratching year head (at least I was) because the documentation on how to bring this together with the older versions of the client is not the directly linked and is not the easiest to find in the guide. For your reference you can find the link to the setup here.

AWS has a v5 and a v3 client software. At this time F5 has been validated with v3 (3.2.2).  During this journey we found that even if you install v5 and then remove it there will still be conflicts with the v3 software. Do not install the v5 client on BIG-IP. 

Deploying an AWS CloudHSM

I recommend that you review the AWS deployment guide.  These are the high level steps that happen in a CloudHSM deployment. 

Create a Cluster -  A cluster is a container into which our hardware HSMs will be deployed. This step is where we deploy a cluster into our AWS VPC and select the subnets.  It is best practice to select a subnet from more than one availability zone (AZ). 

Create one or more HSM - After a cluster is created you need to deploy at least on HSM into it. Best practice is that you deploy one HSM per availability zone AZ in two or more AZs. 

Initialize a Cluster - To initialize the cluster you will need to sign a <cluster ID>_ClusterCsr.csr file with your CA, upload the signed certificate back to the cluster.  This is the step that we will use the certificate from our CA to import on BIG-IP.

Activate the Cluster - Once the cluster is initialized it will need to be activated prior to connecting to it with BIG-IP.  This involved connecting to the cluster via the HSM management client, sending an activate command, changing a password, and the option of creating Crypto User account(s) to be used by HSM clients. My experience was that this was easier to with a Amazon Linux client with SDKv5 to connect to it, activate it on the CLI and create the crypto users (CU) that we will need to configure on BIG-IP.

BIG-IP Licensing for a Network HSM

If you are purchasing BIG-IP on a PAY-AS-YOU-GO instance it is already licensed for a NetworkHSM and able to use CloudHSM.  If your license is procured via either means you may need to contact F5.

Checking the license via TMUI

Checking the license via CLI

[admin@ip-172-31-100-10:Active:Standalone] config # cat bigip.license | grep -i hsm
active module :                  SSL Orchestrator, High Performance VE, 8 vCPU|IZCWUVM-FHYEEZQ|External Interface and Network HSM, VE|APM, Limited|SSL, VE|Max Compression, VE|Exclusive Version, v14.X - 18.X|Anti-Virus Checks|Base Endpoint Security Checks|Firewall Checks|Network Access|Secure Virtual Keyboard|APM, Web Application|Machine Certificate Checks|Protected Workspace|Remote Desktop|App Tunnel
#       License Tokens for Module External Interface and Network HSM, VE key IZCWUVM-FHYEEZQ
pkcs11_nethsm :                  enabled
[admin@ip-172-31-100-10:Active:Standalone] config #

Once you have a license that is capable of supporting a network HSM it is time to install and configure the client software on BIG-IP.

BIG-IP Software Install

If we pick up at the software install we can see the following steps.  After this we refer you to follow the steps to configure the client via the AWS documentation pages.

Installing the software on BIG-IP (you will need to be able to access S3 from your VPC; either over the Internet or via VPC endpoints)

cd /shared/
mkdir nethsm
cd nethsm
 
curl -O https://s3.amazonaws.com/cloudhsmv2-software/CloudHsmClient/EL7/cloudhsm-client-pkcs11-3.2.1-1.el7.x86_64.rpm
curl -O https://s3.amazonaws.com/cloudhsmv2-software/CloudHsmClient/EL7/cloudhsm-client-3.2.1-1.el7.x86_64.rpm

rpm -ivh cloudhsm-client-pkcs11-3.2.1-1.el7.x86_64.rpm
rpm -ivh cloudhsm-client-3.2.1-1.el7.x86_64.rpm

Ok we have the software. Let's take a look at an example config from a working system and then understand how to get from where we are at to where we want to go; using CloudHSM. 

Configuration Files on F5

Installing the software creates files in /opt/cloudhsm/

[admin@ip-172-31-100-10:Active:Standalone] cloudhsm # ls -lah
total 36K
drwxr-xr-x.  9 root    hsmuser 4.0K Jun 30 07:53 .
drwxr-xr-x. 38 root    root    4.0K Aug 17 13:25 ..
drwxr-xr-x.  2 root    root    4.0K Oct  8  2020 bin
lrwxrwxrwx.  1 root    root      10 Jun 30 07:53 daemon -> run/daemon
drwxr-xr-x.  2 root    root    4.0K Jun 30 07:53 data
drwxr-xr-x.  2 root    root    4.0K Jun 30 07:53 doc
drwxr-xr-x.  3 root    root    4.0K Jun 30 08:35 etc
drwxr-xr-x.  2 root    root    4.0K Jun 30 07:53 include
drwxr-xr-x.  2 root    root    4.0K Jun 30 07:53 lib
drwxr-xr-x.  4 hsmuser hsmuser 4.0K Jun 30 07:53 run

We are interested in the cloudhsm/etc directory

[admin@ip-172-31-100-10:Active:Standalone] etc # pwd
/opt/cloudhsm/etc
[admin@ip-172-31-100-10:Active:Standalone] etc # ls -lah
total 40K
drwxr-xr-x. 3 root root     4.0K Jun 30 08:35 .
drwxr-xr-x. 9 root hsmuser  4.0K Jun 30 07:53 ..
drwxr-xr-x. 2 root root     4.0K Jun 30 07:53 certs
-rwxr-xr-x. 1 root root     1.4K Oct  8  2020 client.crt
-rwxr-xr-x. 1 root root     1.7K Oct  8  2020 client.key
-rw-r--r--. 1 root root     1.1K Jun 30 07:56 cloudhsm_client.cfg
-rw-r--r--. 1 root root     1.2K Oct  8  2020 cloudhsm_client.cfg.old
-rw-r--r--. 1 root root      704 Jun 30 07:56 cloudhsm_mgmt_util.cfg
-rw-r--r--. 1 root root      600 Oct  8  2020 cloudhsm_mgmt_util.cfg.old
-rw-r--r--. 1 root webusers 1.4K Jun 30 08:35 customerCA.crt
[admin@ip-172-31-100-10:Active:Standalone] etc #

 In my lab I have an HSM configured at 172.31.10.87; below you can see it listed in the config file.  So let's take a look at what we will need to do to get to a functional setup.  

 

[admin@ip-172-31-100-10:Active:Standalone] etc # cat cloudhsm_mgmt_util.cfg
{
    "scard": {
        "certificate": "cert-sc",
        "enable": "no",
        "pkey": "pkey-sc",
        "port": 2225
    },
    "servers": [
        {
            "CAfile": "",
            "CApath": "/opt/cloudhsm/etc/certs",
            "certificate": "/opt/cloudhsm/etc/client.crt",
            "e2e_encryption": {
                "enable": "yes",
                "owner_cert_path": "/opt/cloudhsm/etc/customerCA.crt"
            },
            "enable": "yes",
            "hostname": "172.31.10.87",
            "name": "172.31.10.87",
            "pkey": "/opt/cloudhsm/etc/client.key",
            "port": 2225,
            "server_ssl": "yes",
            "ssl_ciphers": ""
        }
    ]
}

admin@ip-172-31-100-10:Active:Standalone] etc # cat cloudhsm_client.cfg
{
    "client": {
        "CriticalAlertScript": "",
        "create_object_minimum_nodes": 1,
        "daemon_id": 1,
        "e2e_owner_crt_path": "/opt/cloudhsm/etc/customerCA.crt",
        "log_level": "INFO",
        "reconnect_attempts": -1,
        "reconnect_interval": 3,
        "socket_type": "UNIXSOCKET",
        "sslreneg": 0,
        "tcp_port": 1111,
        "workers": 1,
        "zoneid": 0
    },
    "dualfactor": {
        "certificate": "certificate.crt",
        "dualfactor_ch_ssl_ciphers": "default",
        "dualfactor_ssl": "yes",
        "enable": "no",
        "pkey": "pkey.pem",
        "port": 2225
    },
    "loadbalance": {
        "enable": "yes",
        "prefer_same_zone": "no",
        "relative_idleness_weight": 1,
        "sucess_rate_weight": 1
    },
    "server": {
        "hostname": "172.31.10.87",
        "port": 2223
    },
    "ssl": {
        "CApath": "/opt/cloudhsm/etc/certs",
        "certificate": "/opt/cloudhsm/etc/client.crt",
        "pkey": "/opt/cloudhsm/etc/client.key",
        "server_ch_ssl_ciphers": "default",
        "server_ssl": "yes"
    }

Step 1 - Install all Certificates on BIG-IP 

To connect to the HSM you need to have the certificates installed on the system. 

  • customerCA.crt (provided by you) back in the initialization step.
  • client.crt (default from client software install or custom)
  • client.key (default from client software install or custom)

If you do not want to use the default certificates please look here. 

[admin@ip-172-31-100-10:Active:Standalone] etc # pwd
/opt/cloudhsm/etc
[admin@ip-172-31-100-10:Active:Standalone] etc # ls -lah
total 40K
drwxr-xr-x. 3 root root     4.0K Jun 30 08:35 .
drwxr-xr-x. 9 root hsmuser  4.0K Jun 30 07:53 ..
drwxr-xr-x. 2 root root     4.0K Jun 30 07:53 certs
-rwxr-xr-x. 1 root root     1.4K Oct  8  2020 client.crt
-rwxr-xr-x. 1 root root     1.7K Oct  8  2020 client.key
-rw-r--r--. 1 root root     1.1K Jun 30 07:56 cloudhsm_client.cfg
-rw-r--r--. 1 root root     1.2K Oct  8  2020 cloudhsm_client.cfg.old
-rw-r--r--. 1 root root      704 Jun 30 07:56 cloudhsm_mgmt_util.cfg
-rw-r--r--. 1 root root      600 Oct  8  2020 cloudhsm_mgmt_util.cfg.old
-rw-r--r--. 1 root webusers 1.4K Jun 30 08:35 customerCA.crt
[admin@ip-172-31-100-10:Active:Standalone] etc #

Step 2 - Configure the Software

 In this step we run a command that populates the server IP address into our config files for the cloudhsm_client.cfg and the cloudhsm_mgmt_util.cfg files

/opt/cloudhsm/bin/configure -a 172.31.10.87

Step 3 - Test Connecting to the HSM from BIG-IP

Once you have configured the software you should be able to connect to it. 

[admin@ip-172-31-100-10:Active:Standalone] bin # ./cloudhsm_mgmt_util /opt/cloudhsm/etc/cloudhsm_mgmt_util.cfg
Ignoring E2E enable flag in the configuration file

Connecting to the server(s), it may take time
depending on the server(s) load, please wait...

Connecting to server '172.31.10.87': hostname '172.31.10.87', port 2225...
Connected to server '172.31.10.87': hostname '172.31.10.87', port 2225.
E2E enabled on server 0(172.31.10.87)

aws-cloudhsm>listUsers
Users on server 0(172.31.10.87):
Number of users found:3

    User Id    		User Type	User Name			   MofnPubKey	 LoginFailureCnt	 2FA
         1		CO   		admin                           	 NO               0	 	  NO
         2		AU   		app_user                        	 NO               0	 	  NO
         3		CU   		f5                              	 NO               0	 	  NO


aws-cloudhsm>

You will need a crypto user.  If you did not create one when you initialized and activated the cluster you can do so from BIG-IP.  Please refer to my comments above about how the v5 client (not supported and do not install on BIG-IP) seems more better for this process.  In the example below I create a user from BIG-IP.

aws-cloudhsm>createUser CU F5_2 mypassword
*************************CAUTION********************************
This is a CRITICAL operation, should be done on all nodes in the
cluster. AWS does NOT synchronize these changes automatically with the 
nodes on which this operation is not executed or failed, please 
ensure this operation is executed on all nodes in the cluster.  
****************************************************************

Do you want to continue(y/n)?y
Creating User F5_2(CU) on 1 nodes
createUser success on server 0(172.31.10.87)


aws-cloudhsm>

This is the username that will be used to configure BIG-IP.  Once you have created their user you will need to start (or resart) pkcsd and the cloudhsm-client services.  Please see the F5 guide for more detail. 

Step 4 - Setting up Testing the HSM Connection from TMOS

Prior to setting up a connection to the CLoudHSM you will need to create a crypto user (CU), vendor and partition on BIG-IP. Here is an example of doing so via TMUI and then testing the connection.

 

Or we can run the same test from the CLI
[admin@ip-172-31-100-10:Active:Standalone] config # tmsh run sys crypto nethsm-test   
[Info]: Begin
Note that in both tests you may see an error
 
C_Finalize failed with error CKR_CRYPTOKI_NOT_INITIALIZED : 0x00000190
 This is a non-blocking issue that we are tracking to be fixed. The connection to the HSM is working if you see Pass returned in the results. 

Step 5 - Creating and Inspecting a HSM Certificate on BIG-IP

Now that you have setup and can connect to the HSM you can create certificates on BIG-IP  in the "normal" way that we had in the past and selecting NetworkHSM as the type.

Below we are looking at an existing certificate and we can see the security type of NetHSM.

 

At this point you can reference the certificate in SSL profiles and apply them to virtual servers. 

Learning Along the Way

There are a few things to think about as you deploy AWS CloudHSM to be used with BIG-IP.  

  • You may need to edit the AWS Security Group (SG) associated with the ENI(s) for CloudHSM.  It is common for F5 systems in AWS to have custom security groups and not the default security groups. One option is to add the CloudHSM SG to BIG-IP, and the other is to add rules to the CloudHSM SG.  If you are going to stack SGs on BIG-IP make sure you understand the the implemented state. 
  • Pay close attention when you configure Crypto User information on BIG-IP.  The first time I setup it up I read the instructions as two fields.  User:Password and did not in put the correct information. In TMUI we need to provide three objects: Partition, User, Password. 
    • User = partition (example auto)
    • Password = cryptousername:password
  • If you have setup the user auth correctly restarting the cloudhsm-client.service will show a successful connection  
    • [admin@ip-172-31-100-10:Active:Standalone] ~ # service cloudhsm-client status -l
Redirecting to /bin/systemctl status  -l cloudhsm-client.service
* cloudhsm-client.service - AWS CloudHsm Client daemon
   Loaded: loaded (/usr/lib/systemd/system/cloudhsm-client.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2023-08-18 09:04:24 PDT; 5s ago
     Docs: https://docs.aws.amazon.com/cloudhsm/latest/userguide/introduction.html
 Main PID: 1633 (cloudhsm_client)
   CGroup: /system.slice/cloudhsm-client.service
           `-1633 /opt/cloudhsm/bin/cloudhsm_client /opt/cloudhsm/etc/cloudhsm_client.cfg

Aug 18 09:04:25 ip-172-31-100-10.us-east-2.compute.internal cloudhsm_client[1633]: 2023-08-18T16:04:25Z liquidSecurity INF: check_preferred_srv_status_noclock: New preferred server node id:7
Aug 18 09:04:25 ip-172-31-100-10.us-east-2.compute.internal cloudhsm_client[1633]: 2023-08-18T16:04:25Z liquidSecurity INF: do_e2e_encryption_handshake: Trying to login to server as new server connection is established
Aug 18 09:04:25 ip-172-31-100-10.us-east-2.compute.internal cloudhsm_client[1633]: 2023-08-18T16:04:25Z liquidSecurity INF: e2e_handle_client_request:  Got Authorize session response
Aug 18 09:04:25 ip-172-31-100-10.us-east-2.compute.internal cloudhsm_client[1633]: 2023-08-18T16:04:25Z liquidSecurity INF: get_partition_info: Get pHSM Info using e2e mgmtch
Aug 18 09:04:25 ip-172-31-100-10.us-east-2.compute.internal cloudhsm_client[1633]: 2023-08-18T16:04:25Z liquidSecurity INF: e2e_handle_client_request: Authorize session SUCCESS
Aug 18 09:04:25 ip-172-31-100-10.us-east-2.compute.internal cloudhsm_client[1633]: 2023-08-18T16:04:25Z liquidSecurity INF: e2e_handle_client_request: Got Partition Info
Aug 18 09:04:25 ip-172-31-100-10.us-east-2.compute.internal cloudhsm_client[1633]: 2023-08-18T16:04:25Z liquidSecurity INF: e2e_handle_client_request: GetPartitionInfo success 0 : HSM Return: SUCCESS
Aug 18 09:04:25 ip-172-31-100-10.us-east-2.compute.internal cloudhsm_client[1633]: 2023-08-18T16:04:25Z liquidSecurity INF: e2e_handle_client_request: HSM FIPS STATE 2
Aug 18 09:04:26 ip-172-31-100-10.us-east-2.compute.internal cloudhsm_client[1633]: 2023-08-18T16:04:26Z liquidSecurity INF: libevmulti_init: Initializing events
Aug 18 09:04:26 ip-172-31-100-10.us-east-2.compute.internal cloudhsm_client[1633]: 2023-08-18T16:04:26Z liquidSecurity INF: libevmulti_init: Ready !
[admin@ip-172-31-100-10:Active:Standalone] ~ #
  • The pkcsd service does not start; you need a license that supports NetworkHSM 
    • admin@(ip-172-31-100-10)(cfg-sync Standalone)(Active)(/Common)(tmos.sys.service)# show pkcs11d 
pkcs11d               run (pid 5746) 1 minutes, 3 restarts
admin@(ip-172-31-100-10)(cfg-sync Standalone)(Active)(/Common)(tmos.sys.service)# st

Conclusion

I hope you find this helpful if you need to setup CloudHSM with BIG-IP.  The first time I did this it was a learning process and hopefully others can benefit from the sharing of this experience.  Once you have your CloudHSM deployed bringing a BIG-IP online to use it is not hard. You can use CloudHSM for various F5 offers in AWS such as Advanced WAF, Local Traffic Manager, SSL Orchestrator, of Advanced Firewall Manager. 

Updated Sep 01, 2023
Version 2.0
application delivery
cloud
security
No CommentsBe the first to comment