UPDATE
I’ve made some updates to this design architecture. The new template mimics a real air-gap deployment. The original template below assumed that EIP’s are allowed. However, in most government environments elastic IP’s are not allowed. The original template can still be used but more often in POV/POC scenarios where standing up a HA pair quickly is required.
The new template can achieve the same POC/POV goal but it requires additional steps in how you access the BIG-IP’s. This is because the new template removes elastic IP’s entirely.
Here are the important high level changes.
- The big one here is the removal of EIP’s. In the design below, failover or CFE uses EIP failover. In this new design, failover is achieved by route failover. Where the BIG-IP CFE makes an API call to an AWS endpoint to change the route table. In the other design, the EIP is moved.
- So if EIP’s are not used then how do we get traffic to the active device? Well, im glad you asked. We introduce the concept of an alien IP range. Alien IP’s are an IP space that is foreign to the VPC cidr. We can then create a route in the table and assign this IP to an ENI (active) BIG-IP. This sounds like a hack and it kind of is but its a fully supported method and AWS even references it. Here are a few articles on the subject. AWS Alien IP and F5 Authored Alien IP
- SSM Jump Host is used to access the BIG-IP environment. This is a small linux box used for the purpose of accessing the BIG-IP management and VIP. Since we are not using public IP’s then we need a way in. In a true air-gap network you will have a direct connect setup to your on-prem network and will be able to access from there but for the purpose of this template you have the option to deploy the jump host.
Here is a high level architecture diagram. I encourage you to read the repo deployment guide. There is quite a bit of detail on how this works and the repo is there for you to clone and deploy into your own environment.
The new air-gap guide uses the AWS CLI as this is the path of least resistance in terms of testing and deployment goes. But you definitely can deploy this via the GUI in AWS but some steps may prove more difficult, so for this reason you should have the AWS CLI installed and programmatic access keys to your environment. There are other requirements and these are listed in the repo.
If you have any questions feel free to open a github issue or comment here in the post.
Github Repo link to the guide: f5-aws-cloudformation-v2-govcloud/examples/failover-airgap/AIRGAP-GUIDE.md at main · F5GovSolutions/f5-aws-cloudformation-v2-govcloud · GitHub
The rest of this post covers the original design which is EIP failover.
This guide provides a walk-through for deploying an active/standby F5 BIG-IP High Availability (HA) cluster within AWS GovCloud (US) using the AWS Console.
Code Repository: F5GovSolutions/f5-aws-cloudformation-v2-govcloud
While the repository’s examples/failover/GOVCLOUD-GUIDE.md handles the exact AWS CLI workflows, this document serves as the web console companion.
The Partition Problem
The upstream commercial F5 templates (F5Networks/f5-aws-cloudformation-v2) assume a standard AWS commercial partition. Pointing those unmodified templates at GovCloud breaks deployment for three distinct reasons:
- Partition Isolation (aws-us-gov): GovCloud uses unique ARNs, regional endpoints, and an entirely separate AMI catalog. Furthermore, CloudFormation mandates that a parent stack’s nested templates reside in an S3 bucket within the exact same region and partition. CloudFormation cannot fetch objects across partitions, meaning all templates must be pre-staged in a local GovCloud bucket.
- Air-Gapped Realities: In isolated enclaves with no internet egress, a booting BIG-IP cannot pull the runtime-init installer or Automation Toolchain RPMs from public GitHub repositories or F5 CDNs. Everything must be hosted locally inside an S3 bucket, accessible via VPC Gateway and Interface endpoints.
- The 17.x Clustering Bug: On the BIG-IP 17.x code train, a startup-timing condition occasionally prevents the local device-trust domain (/Common/Root) from initializing properly on first boot. When this occurs, Declarative Onboarding (DO) cannot establish the trust domain or failover group, leaving both nodes deadlocked. This modified template embeds an automated, non-blocking self-heal script to orchestrate the recovery process without manual intervention.
What it deploys
The root template (failover.yaml) orchestrates a series of nested module stacks (network, access, ingress, application, and individual BIG-IP nodes) to build:
- An active/standby pair of BIG-IP Virtual Editions (VE) deployed across two Availability Zones using a 3-NIC topology (Management, External, Internal).
- Cloud Failover Extension (CFE) integration to re-map floating application IPs and AWS routes during a failover event.
- Automated onboarding via F5 BIG-IP Runtime Init, executing Declarative Onboarding (DO) for core network clustering, Application Services 3 (AS3) for virtual servers/WAF policies, and CFE for AWS API integration.
The architecture uses Pay-As-You-Go (PAYG) marketplace licensing and is fully validated on the 3nic-payg…-with-app runtime-init configuration.
Core Architecture Differences vs. Commercial
Staged-Bucket Deployment Mode
Because of partition constraints, your local GovCloud S3 bucket serves as the single source of truth. It must hold two types of data: the nested CloudFormation templates cloned from the repository, and the target installation binaries (the runtime-init .run installer and the accompanying DO/AS3/CFE RPM packages).
The Unauthenticated 403 Pitfall
CloudFormation reads your deployment templates using your active IAM user/role credentials. However, during the bootstrap phase, the individual BIG-IP instances download their configuration files and RPM packages over an unauthenticated HTTPS request. They do not sign these requests with AWS credentials.
If your S3 bucket permissions block public read access completely, the instances receive an HTTP 403 Forbidden response, initialization fails, and CloudFormation rolls back the stack. Because GovCloud blocks public access by default, you must apply a scoped bucket policy allowing s3:GetObject to Principal: “*” paired with a DenyInsecureTransport block.
Single-Toggle VPC Endpoints
Isolated enclaves require four distinct VPC endpoints to function: S3 (Gateway) for fetching installation packages and AS3 WAF policies, and Interface endpoints for EC2, Secrets Manager, and CloudFormation APIs. Missing even one endpoint causes silent automation failures.
To simplify this, the template consolidates these requirements into a single parameter: setting provisionS3Endpoint=true automatically provisions all four regional endpoints with private DNS enabled. Note that your bucket, stack, and endpoints must reside within the same AWS region.
Automated Clustering Self-Heal
To bypass the 17.x initialization bug without manual operator access, the template installs a localized orchestration loop during the pre_onboard hook. The mechanism drops three files onto the file system:
- cluster-heal.sh: The main orchestrator script.
- cluster-heal-trust.py: A native Python script that securely pulls the admin password from Secrets Manager via a SigV4 request using the instance’s IAM role, then invokes the local device trust commands.
- /etc/cron.d/cluster-heal: A cron job that triggers the orchestrator every 3 minutes.
This loop checks the cluster status. If it detects a missing root domain, it triggers a single controlled reboot, reads the peer node’s address from the runtime logs, pulls the password, and establishes device trust.
Once synchronized, it issues the CloudFormation success signal and disables its own cron job. This preserves DO as the declarative source of truth without creating configuration drift. During this process, the CloudFormation stack will remain in CREATE_IN_PROGRESS for roughly 25 to 30 minutes, which is normal behavior.
Elastic IP Allocation Constraints
The default AWS regional quota is 5 Elastic IPs (EIPs). Depending on your topology, this architecture can request up to 7. To prevent deployment failures due to exhausted quotas, use the template toggles to limit public allocations:
- Default Configuration (Public Mgmt, Public VIP, Public Self IPs): 7 EIPs
- Standard Air-Gap Profile (Public Mgmt, Private VIP, Private Self IPs): 4 EIPs
- Strict Isolated Profile (Private Mgmt via Bastion, Private VIP, Private Self IPs): 3 EIPs
Verify your regional limit under Service Quotas → EC2-VPC Elastic IPs (L-0263D0A3) before executing the template.
Deployment Steps (AWS Console)
Before You Begin
Select a single target GovCloud region (us-gov-east-1 or us-gov-west-1). Your staging bucket, CloudFormation stack, EC2 Key Pair, and Secrets Manager secret must share this region to prevent cross-region routing failures in isolated enclaves.
1. Provision the staging S3 bucket
S3 → Create bucket, in your Region. Give it a globally unique name.
[S3 Create bucket, Region selector highlighted.]
2. Stage the templates and artifacts
While file uploads can be handled via the S3 web GUI, staging is more reliably managed via the AWS CLI from a local workstation. Installing the AWS CLI is outside the scope of this article but it’s easy and instructions can be Googled.
Execute the following commands from the root of your cloned repository to sync the template architecture and upload your pre-downloaded F5 binaries:
Download commands:
# F5 BIG-IP Runtime Init installer (note: this repo's tag has NO "v" prefix)
curl -fL -o f5-bigip-runtime-init-2.0.3-1.gz.run \
https://github.com/F5Networks/f5-bigip-runtime-init/releases/download/2.0.3/f5-bigip-runtime-init-2.0.3-1.gz.run
# Declarative Onboarding (DO)
curl -fL -o f5-declarative-onboarding-1.47.0-14.noarch.rpm \
https://github.com/F5Networks/f5-declarative-onboarding/releases/download/v1.47.0/f5-declarative-onboarding-1.47.0-14.noarch.rpm
# Application Services (AS3)
curl -fL -o f5-appsvcs-3.56.0-10.noarch.rpm \
https://github.com/F5Networks/f5-appsvcs-extension/releases/download/v3.56.0/f5-appsvcs-3.56.0-10.noarch.rpm
# Cloud Failover Extension (CFE)
curl -fL -o f5-cloud-failover-2.4.0-0.noarch.rpm \
https://github.com/F5Networks/f5-cloud-failover-extension/releases/download/v2.4.0/f5-cloud-failover-2.4.0-0.noarch.rpm
Sync and upload commands (be sure to be at root of the cloned repo), set the variables accordingly:
# Synchronize the nested CloudFormation infrastructure templates
aws s3 sync ./examples/ "s3://$BUCKET/$PREFIX/" --region "$REGION"
# Upload the core runtime engine and toolchain extensions
aws s3 cp f5-bigip-runtime-init-2.0.3-1.gz.run "s3://$BUCKET/$PREFIX/" --region "$REGION"
aws s3 cp f5-declarative-onboarding-1.47.0-14.noarch.rpm "s3://$BUCKET/$PREFIX/bigip-extensions/" --region "$REGION"
aws s3 cp f5-appsvcs-3.56.0-10.noarch.rpm "s3://$BUCKET/$PREFIX/bigip-extensions/" --region "$REGION"
aws s3 cp f5-cloud-failover-2.4.0-0.noarch.rpm "s3://$BUCKET/$PREFIX/bigip-extensions/" --region "$REGION"
3. Grant anonymous read on the artifacts
In the S3 Console, select your staging bucket and navigate to Permissions. Under Block public access (bucket settings), disable the option to Block public access to buckets and objects granted through new public bucket policies (leave ACL blocks enabled).
Next, apply the following bucket policy to allow the instances to pull down the bootstrap packages securely over HTTPS:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PublicReadGetObject",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws-us-gov:s3:::YOUR-BUCKET-NAME-HERE/*"
},
{
"Sid": "DenyInsecureTransport",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": [
"arn:aws-us-gov:s3:::YOUR-BUCKET-NAME-HERE",
"arn:aws-us-gov:s3:::YOUR-BUCKET-NAME-HERE/*"
],
"Condition": { "Bool": { "aws:SecureTransport": "false" } }
}
]
}
Note: Ensure you replace YOUR-BUCKET-NAME-HERE with your actual bucket name. Do not include any trailing lines or comments in the JSON editor.
[S3 Permissions — Block public access settings.]
4. Store the Admin Password Secret
Navigate to AWS Secrets Manager → Store a new secret. Choose Other type of secret, select the Plaintext tab, clear the default JSON template, and type your raw password string.
Name the secret (e.g., f5-bigip-admin) and save it using the default settings. Copy the resulting Secret ARN (arn:aws-us-gov:secretsmanager:…); both nodes query this specific object at startup.
[Secrets Manager Store secret, plaintext tab.]
[Secrets Manager: Name your secret.]
5. Create the SSH key pair
EC2 → Key Pairs → Create key pair, download the .pem. The key pair is regional. This is optional but recommended you create one before template launch.
[Screenshot: EC2 Create key pair.]
(You can leave the parameter blank in the template and let the stack auto-create one, but then the private key only lives in SSM Parameter Store — pre-creating is simpler if you need to access the boxes vis SSH)
6. Confirm the BIG-IP AMI exists in your Region
The deployment template discovers the target AMI using automated string matching. Ensure that your region has access to the target image by running the following lookup via your terminal, this can also be done via the GUI:
aws ec2 describe-images --region "$REGION" --owners aws-marketplace \
--filters "Name=name,Values=*17.5*PAYG-Best Plus 25Mbps*" \
--query 'reverse(sort_by(Images,&CreationDate))[].[Name,ImageId,CreationDate]' --output table
7. Launch the stack (CloudFormation)
- Open CloudFormation, select your target region, and click Create stack (with new resources).
- Select Template is ready, choose Amazon S3 URL, and paste the direct object URL to your staged failover.yaml file.
- Configure the mandatory parameters:
- Stack Name: Define an environment-specific identifier.
- S3 Configurations: Populate s3BucketName and s3BucketRegion.
- Secrets Management: Paste your copied Secrets Manager ARN into bigIpSecretArn.
- Security Access: Set restrictedSrcAddressMgmt and restrictedSrcAddressApp to your administrative CIDR blocks to avoid exposing management interfaces.
- Air-Gap Toggles (For isolated VPCs): Change provisionS3Endpoint to true, and flip both provisionPublicIpExternalSelf and provisionPublicIpVip to false.
- In the options screen, consider setting Stack failure options to Preserve successfully provisioned resources during your initial test runs. This keeps instances alive for log inspection if a deployment fails.
- Acknowledge the CAPABILITY_NAMED_IAM prompt and click Submit.
[CloudFormation Specify template — Amazon S3 URL field.]
[Parameters page, air-gap toggles section.]
[Review — Acknowledgement and Stack failure options.]
8. Wait it out
Expect ~25–30 minutes in CREATE_IN_PROGRESS while the self-heal forms the cluster, then CREATE_COMPLETE. This is normal — it is not stuck. After 10 minutes, go ahead and admin SSH into the boxes, they should accept your admin secret by now, if not use your SSH key you created and referenced in the template. See the commands below or in the git repo guide.
Validating the deployment
From the stack Outputs, grab a management IP, then:
- **Onboarding:**grep -i ‘All operations completed successfully’ /var/log/cloud/bigIpRuntimeInit.log; the prompt should show failover01.local / failover02.local, not ip-x-x-x-x.
- **Clustering:**tmsh show cm sync-status → In Sync (green), Mode: high-availability, one node Active and one Standby.
- **CFE:**GET https://localhost/mgmt/shared/cloud-failover/inspect returns a populated object (instance, addresses, trafficGroup).
- The self-heal’s own narration lives in /config/cluster-heal/log — the primary place to look if the stack runs long; you’ll see the reboot → add-to-trust → creating failoverGroup → cluster In Sync → cfn-signal sent OK → disabling self-heal sequence.
If clustering genuinely never forms (the rare case where the self-heal exhausts its attempts), the repo guide has a validated manual fallback that follows the same reboot → rebuild trust → reapply sequence.
What’s GovCloud-ready today — and what isn’t
Let’s be clear about scope. The GovCloud template— staged-bucket defaults, the four VPC endpoints, the clustering self-heal, the bumped extensions — currently applies to one solution: the failover active/standby pair on the 3-NIC PAYG -with-app runtime-init config. The repo’s autoscale, the quickstart standalone, and the other failover variants (2-NIC, BYOL, and the non–with-app configs) still carry upstream configuration and are not yet GovCloud-validated. Adapting them is planned work and would follow exactly the pattern documented here.
And the self-heal is a workaround for an F5 platform bug, not a permanent fix. The right long-term resolution is the platform defect behind the KB; when a fixed build is in play, the self-heal simply stays inert.
Where to go next
- The repo:F5GovSolutions/f5-aws-cloudformation-v2-govcloud
- Full CLI walkthrough (first-time operator):examples/failover/GOVCLOUD-GUIDE.md
- Parameter and architecture reference:examples/failover/README.md
- The self-heal sources:examples/failover/bigip-configurations/cluster-heal.sh and cluster-heal-trust.py
If you’re standing up BIG-IP HA in an enclave, start from the 4-EIP air-gap profile with provisionS3Endpoint=true, stage everything in one in-Region bucket, and let the self-heal do the clustering. File issues on the repo if you hit something the guide doesn’t cover.