Deploy BIG-IP in AWS with HA across AZ’s - without using EIP’s
Background:
The CloudFormation templates that are provided and supported by F5 are an excellent resource for customers to deploy BIG-IP VE in AWS. Along with these templates, documentation guiding your F5 deployment in AWS is an excellent resource.
And of course, DevCentral articles are helpful. I recommend reading about HA topologies in AWS to start. I hope my article today can shed more light on an architecture that will suit a specific set of requirements: No Elastic IP's (EIP’s), High Availability (HA) across AZ’s, and multiple VPC’s.
Requirements behind this architecture choice:
I recently had a requirement to deploy BIG-IP appliances in AWS across AZ’s. I had read the official deployment guide, but I wasn’t clear on how to achieve failover without EIP’s. I was given 3 requirements:
- HA across AZ’s. In this architecture, we required a pair of BIG-IP devices in Active/Standby, where each device was in a different AZ. I needed to be able to fail over between devices.
- No EIP’s. This requirement existed because a 3rd party firewall was already exposed to the Internet with a public IP address. That firewall would forward inbound requests to the BIG-IP VE in AWS, which in turn proxied traffic to a pair of web servers. Therefore, there was no reason to associate an EIP (a public IP address) with the BIG-IP interface. In my demo below I have not exposed a public website through a 3rd party firewall, but to do so is a simple addition to this demo environment.
- Multiple VPC’s. This architecture had to allow for multiple VPC’s in AWS. There was already a “Security VPC” which contained firewalls, BIG-IP devices, and other devices, and I had to be able to use these devices for protecting applications that were across 1 or more disparate VPC’s.
Meeting the requirements:
HA across AZ’s
This is the easiest of the requirements to meet because F5 has already provided templates to deploy these in AWS. I personally used a 2-nic device, with a BYOL license, deployed to an existing VPC, so that meant my template was this one. After this deployment is complete, you should have a pair of devices that will sync their configuration.
At time of failover
The supported F5 templates will deploy with the Advanced HA iApp. It is important that you configure this iApp after you have completed your AWS deployments. The iApp uses IAM permissions deployed with the template to make API calls to AWS at the time of failover. The API calls will update the route tables that you specify within the iApp wizard. Because this iApp is installed on both devices, either device can update the route in your route tables to point to its own interface.
Update as of Dec 2019
This article was first written Feb 2019, and in Dec 2019 F5 released the Cloud Failover Extension (CFE), which is a cloud-agnostic, declarative way to configure failover in multiple public clouds. You can use the CFE instead of the Advanced HA iApp to achieve high availability between BIG-IP devices in cloud.
Update as of Apr 2020
Your API calls will typically be sent to the public Internet endpoints for AWS EC2 API calls. Optionally, you can use AWS VPC endpoints to keep your API calls out to AWS EC2 from traversing the public Internet. My colleague Arnulfo Hernandez has written an article explaining how to do this.
No EIP’s
Configure an “alien IP range”
I’m recycling another DevCentral solution here. You will need to choose an IP range for your VIP network that does not fall within the CIDR that is assigned to your VPC. Let’s call it an “alien range” because it “doesn’t belong” in your VPC and you couldn’t assign IP addresses from this range to your AWS ENI’s. Despite that, now create a route table within AWS that points this “alien range” to your Active BIG-IP device’s ENI (if you’re using a 2+ nic device, point it to the correct data plane NIC, not the Mgmt. interface). Don’t forget to associate the route table with specific subnets, per your design. Alternatively, you could add this route to the default VPC route table.
Create a VIP on your active device
Now create a VIP on your active device and configure the IP address as an IP within your alien range. Ensure the config replicates to your standby device. Ensure that source/destination checking is disabled on the ENI’s that your AWS routes are pointing to (on both Standby and Active devices). You should now have a VIP that you can target from other hosts in your VPC, provided that the route you created above is applied to the traffic destined to the VIP.
Multiple VPC’s
For extra credit, we’ll set up a Transit Gateway. This will allow other VPCs to route traffic to this “alien range” also, provided that the appropriate routes exist in the remote VPC’s, and also are applied to the appropriate Transit Gateway Route Table. Again, I’m recycling ideas that have already been laid out in other DevCentral posts.
I won’t re-hash how to set up a transit gateway in AWS, because you can follow the linked post above. Sufficed to say, this is what you will need to set up if you want to route between multiple VPC’s using a transit gateway:
- 2 or more VPC’s
- A transit gateway in AWS
- Multiple transit gateway attachments that attach the transit gateway and each VPC you would like to route between. You will need one attachment per VPC.
- A transit gateway route table that is associated with each attachment.
I will point out that you need to add a route for your “alien range” in your transit gateway route table, and in your remote VPC’s. That way, hosts in your remote VPC’s will forward traffic destined to your alien range (VIP network) to the transit gateway, and the transit gateway will forward it to your VPC, and the route you created in Step A will forward that traffic to your active BIG-IP device.
Completed design:
After the above configuration, you should have an environment that looks like the diagram below:
Tips
Internet access for deployments: When you deploy your BIG-IP devices, they will need Internet access to pull down some resources, including the iApp. So if you are deploying devices into your existing VPC, make sure you have a reachable Internet Gateway in AWS so that the devices have Internet access through both their management interface, and their data plane interface(s).
Internet access for failover: Remember that an API call to AWS will still use an outbound request to the Internet. Make sure you allow the BIG-IP devices to make outbound connections over HTTPS to the Internet. If this is not available, you will find that your route tables are not updated at time of failover. (If you have a hard requirement that your devices should not have outbound access to internet, you can follow Arnulfo's guide linked to above and use VPC endpoints to keep this traffic on your local VPC)
iApp logs: you can enable this in the iApp settings. I have used the logs (in /var/ltm/log) to troubleshoot issues that I have created myself. That’s how I learned not to accidentally cut off Internet access for your devices!
Don’t forget about return routes if SNAT is disabled: Just like on-prem environments, if you disable SNAT, the pool member will need to route return traffic back to the BIG-IP device. You will commonly set up a default route (0.0.0.0/0) in AWS, point it at the ENI of the active BIG-IP device, and associate this route table with the subnet containing the pool members. If the pool members are in a remote VPC, you will need to create this route on the transit gw route table also.
Don’t accidentally cut off internet access: When you configure the default route of 0.0.0.0/0 to point to eth1 of the BIG-IP device, don’t apply this route to every subnet in your Security VPC. It may be easy to do so accidentally, but remember that it could cause the API calls to update route tables to fail when the Standby device becomes Active.
Don’t forget to disable source/dest check on your ENI’s. This is configured by the template, but if you have other devices that require it, remember to check this setting.
hi @Michael OLeary
So let me get this straight...when i setup HA across AZ's for the purpose of load balancing internal services within a VPC. Failover wont work if my F5's dont have internet access?
- MichaelOLearyEmployee
Hi Sebastian. Failover of the devices will work without Internet access. The standby device will become active at the time of failover. However, if you'd like to have an AWS route automatically updated with an API call (which can be easily set up with the HA iApp), then you'll need the BIG-IP to be able to make an HTTPS connection to the (public) AWS API endpoint. The alternative would be to script this route update yourself for your failover scenario. Does that make sense?
Thanks Michael thats what i thought. Makes sense.
HI Michael, one more question. So the F5 will failvoer.. (great .. thats normal)... Let me know if you can shed some light, how would i go about scripting this? would i do something on the f5? or would i use another server that requires access ? DM if you want :D
- MichaelOLearyEmployee
Hi Sebastian, no problem. Likewise feel free to DM me. Yes, the F5 failover part is normal, and yes, updating any AWS routes automatically will require an API call. If you don't want to let the HA iApp set that up for you, you could script yourself. Consider this out of scope of this article, but you'll need to consider: a) you need to make a AWS API call to update the specific routes you have b) you can do this using a AWS cli command from a Windows or Linux box that has Internet access c) you should have some failure detection mechanism so that you can run this script when it is required. Obviously it's easier to let the HA iApp set this up, but if you have other non-F5 things to do at the time of failover, like update your application, then you could script this as part of your whole recovery activity. However, if you don't already use the AWS API for your own processes, I would recommend just using the HA iApp, especially to start with as you learn.
Cool, thanks .. last question :D can i use the API Gateway to handle this? (Does that make sense) ?
One more :D .. can i use aws cloud watch to watch the failure of the f5 and when a failure occurs use a lamda function to call the api to failover the routes ?
- MichaelOLearyEmployee
Hi Sebastian, I don't see why not, but I'd consider that beyond the recommendations of this article, because I haven't tested it. I like the idea though, getting those native services to do it instead of a different server. This is all to avoid allowing the BIG-IP appliance to access the internet and make the API call, right? Perhaps you could have the BIG-IP initiate that at failover, but use your own tools? If possible, could you research that and let us know how it works out? I'd be very interested in seeing that set up with tools that allowed you to make that AWS update without having the BIG-IP access the Internet via API call. I think for supportability though, the iApp is still the best bet.
- Wildzfor_386457Nimbostratus
Great, info.. I'm working on deploy BIG-IP in AWS with HA across AZ’s with EIP’s in my case. Do you have any details around how to get routing setup and configured between F5 and the backed servers in AWS?
Thanks,
- MichaelOLearyEmployee
Hi Wildzfor, if you are going to use EIPs, you still use a supported template and the HA iApp. Unless you SNAT all of your traffic to your backend servers, you are going to need a route in AWS so that return traffic from the backend servers routes out of the same BIG-IP device that it came in through. That's where the HA iApp can update it automatically for you. Follow the documentation and let us know how you go.