Deploy Failover BIG-IP Cluster with Existing Stack in Google Cloud using v2 Templates

In this article, I plan to cover how-to deploy a pair of BIG-IP devices with failover and high availability features into a Google Cloud existing network stack. Review other articles in this series to see alternative deployment options.

How-to v2 Failover Example: BIG-IP Cluster on Google Cloud with Existing Network Stack

The following section will be a walkthrough with steps to deploy two BIG-IP instances in GCP using the Failover example with an existing network stack. When complete, the BIG-IP devices will form an Active/Standby cluster. You can use the same techniques in order to copy the templates, modify as needed, and deploy in your environment.

During a failover event on the BIG-IP devices, the new active device (was standby) will detect the problem via heartbeats. The new active BIG-IP device sends API calls to Google API to move objects like alias IP ranges, forwarding rule IP mappings (aka public IPs), and more. This API failover method is thanks to the magic in the F5 Cloud Failover Extension, part of the F5 Automation Toolchain.

Clone GitHub Repository

1. Navigate to GitHub repo for v2 BIG-IP GDM templates

2. Select the "Code" button and copy the HTTPS address

3. Open a terminal (ex. Visual Studio Code) and clone the repository

cd ~/Desktop
git clone https://github.com/F5Networks/f5-google-gdm-templates-v2.git

Create Secret in GCP Secret Manager

One prerequisite for this deployment is to use a secret from GCP Secret Manager. Secret Manager stores API keys, passwords, certificates, and other sensitive data. The secret will be used for the BIG-IP administrator password during onboarding.

1. From your terminal, create a file called password.txt containing the BIG-IP password in one line.

2. Create the secret with gcloud using the password.txt file

Note: For demo purposes of "existing stack", we will use a custom secret name and modify the BIG-IP onboarding files to match.

gcloud secrets create giroux-bigip-secret --data-file="password.txt"

# Sample Output
Created version [1] of the secret [giroux-bigip-secret]

Modify Parameters - Existing Stack

For this section, I will deploy the BIG-IP with an existing network stack in GCP. I customize the example Runtime Init YAML file from GitHub and then copy it to my GCP Bucket with public access. This allows the BIG-IP to retrieve it during onboard. The result will be the following:

Using my existing network stack with 3x VPCs: mgmt, external, internal
2x BIG-IP instance, version=16.1.2.2
4x CPU, 3-NICs
Using my customized runtime-init-conf-3nic-payg-instance01-jeff.yaml and runtime-init-conf-3nic-payg-instance02-jeff.yaml from my GCP Bucket
BIG-IP cluster and network settings only

Here we go!

Note: For demo purposes of this article, we will use a tagged version of the repo to maintain article consistency with code.

1. From your terminal, set DEPLOYMENT_NAME, set CONFIG_FILE, change folders, and change branch to pinned tag

DEPLOYMENT_NAME=giroux-bigip-failover-existing
CONFIG_FILE=sample_failover_existing_network.yaml

cd f5-google-gdm-templates-v2/examples/failover
git checkout tags/v2.4.0.0

2. Review the entire README for instructions and prerequisites

3. Edit the file sample_failover_existing_network.yaml

4. Modify the BIG-IP addresses to match your existing VPC network

Note: My existing network setup consists of the following: mgmt = 10.1.1.0/24, external = 10.1.10.0/24, internal = 10.1.20.0/24. Match your environment and use available IPs within the subnet range. I will also update the cfeTag.

# sample_failover_existing_network.yaml - BIG-IP Failover Cluster with existing stack
# Note: Commented and some Optional lines were removed from
#       the yaml below. This keeps the code block small
#       for easier illustration purposes.

imports:
  - path: failover-existing-network.py
  - path: ../modules/access/access.py
  - path: ../modules/bigip-standalone/bigip_standalone.py
  - path: ../modules/dag/dag.py
resources:
  - name: failover-py
    properties:
      bigIpExternalSelfIp01: 10.1.10.101
      bigIpExternalSelfIp02: 10.1.10.102
      bigIpInternalSelfIp01: 10.1.20.101
      bigIpInternalSelfIp02: 10.1.20.102
      bigIpMgmtAddress01: 10.1.1.101
      bigIpMgmtAddress02: 10.1.1.102
      bigIpPeerAddr: 10.1.10.101
      bigIpImageName: f5-bigip-16-1-2-2-0-0-28-payg-best-plus-1gbps-220505081153
      bigIpRuntimeInitConfig01: >-
        https://storage.googleapis.com/giroux-public/runtime-init-conf-3nic-payg-instance01-jeff.yaml
      bigIpRuntimeInitConfig02: >-
        https://storage.googleapis.com/giroux-public/runtime-init-conf-3nic-payg-instance02-jeff.yaml
      cfeTag: giroux_bigip_ha
      cfeBucket: cfe-storage
      networks:
        mgmtNetworkName: jgiroux-net-mgmt
        externalNetworkName: jgiroux-net-ext
        internalNetworkName: jgiroux-net-int
      subnets:
        mgmtSubnetName: jgiroux-subnet-mgmt
        appSubnetName: jgiroux-subnet-ext
        internalSubnetName: jgiroux-subnet-int
      owner: giroux
      provisionPublicIp: true
      region: us-west1
      restrictedSrcAddressApp: [0.0.0.0/0]
      restrictedSrcAddressMgmt: [0.0.0.0/0]
      uniqueString: giroux-ha
      zone: us-west1-a
    type: failover-existing-network.py

5. Edit the runtime-init-conf-3nic-payg-instance01.yaml and runtime-init-conf-3nic-payg-instance02.yaml files according to your existing network stack. If you changed the cfeTag or secretID in earlier steps, then update the values here too.

6. Save both files and upload to a location accessible by BIG-IP during onboard

# runtime-init-conf-3nic-payg-instance01.yaml and
# runtime-init-conf-3nic-payg-instance02.yaml - sourced from GitHub examples
# My example change...
# secretID
    secretProvider:
      environment: gcp
      type: SecretsManager
      version: latest
      secretId: giroux-bigip-secret

# cfeTag
        externalStorage:
          scopingTags:
            f5_cloud_failover_label: giroux_bigip_ha
        failoverAddresses:
          enabled: true
          scopingTags:
            f5_cloud_failover_label: giroux_bigip_ha

Deploy the BIG-IP Failover Cluster - Existing Stack

1. From your terminal, launch the GDM template

gcloud deployment-manager deployments create ${DEPLOYMENT_NAME} --config ${CONFIG_FILE}

The fingerprint of the deployment is b'XOHySk6KEgmLFJeHLnO70g=='
Waiting for create [operation-1660327289789-5e60f12519bf8-98f7420d-c81fc548]...done.                       
Create operation operation-1660327289789-5e60f12519bf8-98f7420d-c81fc548 completed successfully.
NAME                          TYPE                                                 STATE      ERRORS  INTENT
giroux-ha-app-int-fw      compute.v1.firewall  COMPLETED  []
giroux-ha-app-vip-fw      compute.v1.firewall  COMPLETED  []
giroux-ha-bigip-iam-bind  gcp-types/cloudresourcemanager-
v1:virtual.projects.iamMemberBinding  COMPLETED  []
giroux-ha-bigip-sa        iam.v1.serviceAccount  COMPLETED  []
giroux-ha-bigip-vm-01     compute.v1.instance  COMPLETED  []
giroux-ha-bigip-vm-01-ti  compute.v1.targetInstance  COMPLETED  []
giroux-ha-bigip-vm-02     compute.v1.instance  COMPLETED  []
giroux-ha-bigip-vm-02-ti  compute.v1.targetInstance  COMPLETED  []
giroux-ha-cfe-storage     storage.v1.bucket  COMPLETED  []
giroux-ha-fr-01           compute.v1.forwardingRule  COMPLETED  []
giroux-ha-ha-fw           compute.v1.firewall  COMPLETED  []
giroux-ha-http-hc         compute.v1.healthCheck  COMPLETED  []
giroux-ha-https-hc        compute.v1.healthCheck  COMPLETED  []
giroux-ha-mgmt-fw         compute.v1.firewall  COMPLETED  []
giroux-ha-public-ip-01    compute.v1.address  COMPLETED  []
giroux-ha-tcp-hc          compute.v1.healthCheck  COMPLETED  []
girouxhabigipaccessrole   gcp-types/iam-v1:projects.roles  COMPLETED  []

Validating the Deployment

See the "Validating the Deployment" section or the "Further Exploring" section in the Failover README file for more validation commands.

1. Retrieve the parent template outputs

2. Copy the values for bigIpManagementPublicIp1, bigIpManagementPublicIp2, and vip1PublicIp

gcloud deployment-manager manifests describe --deployment=${DEPLOYMENT_NAME} --format="value(layout)" | yq .resources[0].outputs

# Sample Output
...snippet
- finalValue: 35.199.156.64
  name: bigIpManagementPublicIp1

- finalValue: 34.82.195.63
  name: bigIpManagementPublicIp2

...snippet...
- finalValue: 35.233.137.160
  name: vip1PublicIp

SSH to BIG-IP Cluster and Review Logs

1. Access the BIG-IP devices and enter bash mode (value from bigIpManagementPublicIp1 and bigIpManagementPublicIp2)

ssh admin@35.199.156.64 -i ~/.ssh/id_rsa

admin@(giroux-ha-bigip1)(cfg-sync In Sync)(Active)(/Common)(tmos)# bash
[admin@giroux-ha-bigip1:Active:In Sync] ~ #

2. Review BIG-IP Runtime Init onboard logs

[admin@giroux-ha-bigip1:Active:Standalone] ~ # cat /var/log/cloud/bigIpRuntimeInit.log

# Sample Output
...snippet...
2022-08-11T19:04:42.573Z [3752]: info: Successfully validated declaration
2022-08-11T19:04:42.661Z [3752]: info: Resolving parameters
2022-08-11T19:04:45.259Z [3752]: info: Executing install operations.
2022-08-11T19:05:15.323Z [3752]: info: Executing service operations.
2022-08-11T19:05:15.369Z [3752]: info: Creating - do 1.31.0 ....
2022-08-11T19:08:38.510Z [3752]: info: Creating - cf 1.11.0 ....
2022-08-11T19:08:52.323Z [3752]: info: Creating - as3 3.38.0 ....
2022-08-11T19:09:25.127Z [3752]: info: Creating - do 1.31.0 ....
2022-08-11T19:10:07.330Z [3752]: info: Initializing telemetryClient
2022-08-11T19:10:08.193Z [3752]: info: Sending f5-teem report
2022-08-11T19:10:08.197Z [3752]: info: All operations finished successfully
2022-08-11T19:10:13.570Z [3752]: info: Successfully sent data to F5 Teem

Testing the Application and WAF

The following tests will work if you deploy the BIG-IP with Layer 4-7 settings. For my demo settings, the "existing stack" does not create BIG-IP application objects nor does it create an application server. Fear not! You can still POST an AS3 declaration to the BIG-IP, or you can login manually and create a pool and listener. Check out the F5 AS3 Docs for more example declarations!

1. From your local machine, curl the demo application (value from vip1PublicIp)

curl http://35.233.137.160 -I

# Sample Output
HTTP/1.1 200 OK
...snippet...
Set-Cookie: BIGipServer~Tenant_1~Shared~Shared_Pool=xxxx; path=/; Httponly
Set-Cookie: TS01aa0884=xxxx; Path=/

2. Perform a security violation by sending a disallowed 'method' of DELETE

curl http://35.233.137.160 -X DELETE

# Sample Output
<html><head><title>Request Rejected</title></head><body>The requested URL was rejected. Please consult with your administrator.<br><br>Your support ID is: 9487250596978229314<br><br><a href='javascript:history.back();'>[Go Back]</a></body></html>

Testing Failover

Let's validate a few things and then force a failover.

1. From your terminal, run gcloud to list the forwarding rule and verify the target BIG-IP instance (1 or 2)

gcloud compute forwarding-rules list --filter="name=(giroux-ha-fr-01)"

# Sample Output
NAME                 REGION    IP_ADDRESS      IP_PROTOCOL  TARGET
giroux-ha-fr-01  us-west1  35.233.137.160  TCP          us-west1-a/targetInstances/giroux-ha-bigip-vm-01-ti

The output shows vm-01 which is BIG-IP instance 1. This matches the CLI on each BIG-IP in which instance 1 shows 'active' and instance 2 shows 'standby'.

# From instance 1
[admin@giroux-ha-bigip-vm-01:Active:In Sync] ~ # 

# From instance 2
[admin@giroux-ha-bigip-vm-02:Standby:In Sync] ~ #

Now, let's force a failover.

2. From the standby BIG-IP, start a 'tail -f' on the restnoded log file to watch live logs

[admin@giroux-ha-bigip-vm-02:Standby:In Sync] ~ # tail -f /var/log/restnoded/restnoded.log

3. From the active BIG-IP, force a failover

[admin@giroux-ha-bigip-vm-01:Active:In Sync] ~ # tmsh run sys failover standby
[admin@giroux-ha-bigip-vm-01:Standby Sync]

4. Go back to the other BIG-IP instance...now 'active' and review the restnoded logs

# Sample Output
...
Thu, 11 Aug 2022 19:46:19 GMT - info: [f5-cloud-failover] Performing Failover - update
Thu, 11 Aug 2022 19:46:19 GMT - fine: [f5-cloud-failover] Address discovery: ....
...
...
Thu, 11 Aug 2022 19:46:19 GMT - finest: [f5-cloud-failover] Updating forwarding rule: giroux-ha-fr-01 to target: xxxx/giroux-ha-bigip-vm-02-ti
Thu, 11 Aug 2022 19:46:19 GMT - info: [f5-cloud-failover] Disassociate NICs successful.
Thu, 11 Aug 2022 19:46:19 GMT - info: [f5-cloud-failover] Associate NICs successful.
Thu, 11 Aug 2022 19:46:20 GMT - finest: [f5-cloud-failover] updateFwdRule operation name: operation-xxxx
Thu, 11 Aug 2022 19:46:45 GMT - info: [f5-cloud-failover] Updated forwarding rules successfully
...
Thu, 11 Aug 2022 19:46:45 GMT - info: [f5-cloud-failover] Failover Complete
Thu, 11 Aug 2022 19:46:48 GMT - finest: [f5-cloud-failover] Download stateFile: ....

[admin@giroux-ha1122-bigip-vm-02:Active:In Sync] ~ #

The log entries validate the forwarding rule was updated to target instance 2.

5. From your terminal, run gcloud to list the forwarding rules again to validate a new target

gcloud compute forwarding-rules list --filter="name=(giroux-ha-fr-01)"

# Sample Output
NAME                 REGION    IP_ADDRESS      IP_PROTOCOL  TARGET
giroux-ha-fr-01  us-west1  35.233.137.160  TCP          us-west1-a/targetInstances/giroux-ha-bigip-vm-02-ti

Test traffic again (see earlier sections) and everything should still work like magic!

Delete and Clean Up

1. Delete the deployment

gcloud deployment-manager deployments delete ${DEPLOYMENT_NAME} -q

Summary

This article shows you how to use the BIG-IP Cloud Solution Templates to easily deploy services and BIG-IP in Google Cloud. I use sample configurations in this article to deploy two BIG-IP devices in a failover cluster with an existing network stack. I then show you how to validate the deployment and review onboard logs. This Failover example is an easy way to have your applications protected by BIG-IP running in HA on Google Cloud. Make sure to check out the other examples in the GitHub repository for scenarios like standalone or autoscale.