on 21-Feb-2019 05:00
The CloudFormation templates that are provided and supported by F5 are an excellent resource for customers to deploy BIG-IP VE in AWS. Along with these templates, documentation guiding your F5 deployment in AWS is an excellent resource.
And of course, DevCentral articles are helpful. I recommend reading about HA topologies in AWS to start. I hope my article today can shed more light on an architecture that will suit a specific set of requirements: No Elastic IP's (EIP’s), High Availability (HA) across AZ’s, and multiple VPC’s.
I recently had a requirement to deploy BIG-IP appliances in AWS across AZ’s. I had read the official deployment guide, but I wasn’t clear on how to achieve failover without EIP’s. I was given 3 requirements:
This is the easiest of the requirements to meet because F5 has already provided templates to deploy these in AWS. I personally used a 2-nic device, with a BYOL license, deployed to an existing VPC, so that meant my template was this one. After this deployment is complete, you should have a pair of devices that will sync their configuration.
The supported F5 templates will deploy with the Advanced HA iApp. It is important that you configure this iApp after you have completed your AWS deployments. The iApp uses IAM permissions deployed with the template to make API calls to AWS at the time of failover. The API calls will update the route tables that you specify within the iApp wizard. Because this iApp is installed on both devices, either device can update the route in your route tables to point to its own interface.
This article was first written Feb 2019, and in Dec 2019 F5 released the Cloud Failover Extension (CFE), which is a cloud-agnostic, declarative way to configure failover in multiple public clouds. You can use the CFE instead of the Advanced HA iApp to achieve high availability between BIG-IP devices in cloud.
Your API calls will typically be sent to the public Internet endpoints for AWS EC2 API calls. Optionally, you can use AWS VPC endpoints to keep your API calls out to AWS EC2 from traversing the public Internet. My colleague Arnulfo Hernandez has written an article explaining how to do this.
I’m recycling another DevCentral solution here. You will need to choose an IP range for your VIP network that does not fall within the CIDR that is assigned to your VPC. Let’s call it an “alien range” because it “doesn’t belong” in your VPC and you couldn’t assign IP addresses from this range to your AWS ENI’s. Despite that, now create a route table within AWS that points this “alien range” to your Active BIG-IP device’s ENI (if you’re using a 2+ nic device, point it to the correct data plane NIC, not the Mgmt. interface). Don’t forget to associate the route table with specific subnets, per your design. Alternatively, you could add this route to the default VPC route table.
Now create a VIP on your active device and configure the IP address as an IP within your alien range. Ensure the config replicates to your standby device. Ensure that source/destination checking is disabled on the ENI’s that your AWS routes are pointing to (on both Standby and Active devices). You should now have a VIP that you can target from other hosts in your VPC, provided that the route you created above is applied to the traffic destined to the VIP.
For extra credit, we’ll set up a Transit Gateway. This will allow other VPCs to route traffic to this “alien range” also, provided that the appropriate routes exist in the remote VPC’s, and also are applied to the appropriate Transit Gateway Route Table. Again, I’m recycling ideas that have already been laid out in other DevCentral posts.
I won’t re-hash how to set up a transit gateway in AWS, because you can follow the linked post above. Sufficed to say, this is what you will need to set up if you want to route between multiple VPC’s using a transit gateway:
I will point out that you need to add a route for your “alien range” in your transit gateway route table, and in your remote VPC’s. That way, hosts in your remote VPC’s will forward traffic destined to your alien range (VIP network) to the transit gateway, and the transit gateway will forward it to your VPC, and the route you created in Step A will forward that traffic to your active BIG-IP device.
After the above configuration, you should have an environment that looks like the diagram below:
Internet access for deployments: When you deploy your BIG-IP devices, they will need Internet access to pull down some resources, including the iApp. So if you are deploying devices into your existing VPC, make sure you have a reachable Internet Gateway in AWS so that the devices have Internet access through both their management interface, and their data plane interface(s).
Internet access for failover: Remember that an API call to AWS will still use an outbound request to the Internet. Make sure you allow the BIG-IP devices to make outbound connections over HTTPS to the Internet. If this is not available, you will find that your route tables are not updated at time of failover. (If you have a hard requirement that your devices should not have outbound access to internet, you can follow Arnulfo's guide linked to above and use VPC endpoints to keep this traffic on your local VPC)
iApp logs: you can enable this in the iApp settings. I have used the logs (in /var/ltm/log) to troubleshoot issues that I have created myself. That’s how I learned not to accidentally cut off Internet access for your devices!
Don’t forget about return routes if SNAT is disabled: Just like on-prem environments, if you disable SNAT, the pool member will need to route return traffic back to the BIG-IP device. You will commonly set up a default route (0.0.0.0/0) in AWS, point it at the ENI of the active BIG-IP device, and associate this route table with the subnet containing the pool members. If the pool members are in a remote VPC, you will need to create this route on the transit gw route table also.
Don’t accidentally cut off internet access: When you configure the default route of 0.0.0.0/0 to point to eth1 of the BIG-IP device, don’t apply this route to every subnet in your Security VPC. It may be easy to do so accidentally, but remember that it could cause the API calls to update route tables to fail when the Standby device becomes Active.
Don’t forget to disable source/dest check on your ENI’s. This is configured by the template, but if you have other devices that require it, remember to check this setting.
Hi Sebastian, yes - but you have to use a Transit Gateway to route between your VPCs. Transit Gateways were released in late 2018, and they will allow you to route between multiple VPCs. Prior to this, you needed to manage VPC peering relationships individually, and these peering relationships did not allow you to route a non-VPC IP range. (The other option prior to the release of Transit Gateways was to build IPSec VPN connections yourself between VPCs and I have not covered that here).
Thanks Michael.
Yes i didnt want to go down the IPsec vpn route.
I need to test this out ASAP 😄
Hello Michael! Thanks for the article. Quick question. Since bigip-1 and bigip-2 are on different AZs, and that means different subnets, how did you solve the default route after you clustered? Not the management routes, the TMOS routes (because they sync these stuff)... Cheers, Rafael!
Hi Rafael, I deployed this template from F5's supported sample templates. I recommend doing the same. The template deployment configures default routes into a LOCAL_ONLY partition on each device, so that this default route for a device does not sync to the paired device.
Hello Michael, thank you for the article, it is really interesting and very useful for us because our desgin don't include EIPs. In case, F5 can't connect to the Internet, do you have workaround for this situation ? Br
Hi Amintej, glad to hear it's useful. 2 things I can think to say in response: 1. If using the template I did, you will need internet connectivity from the BIG-IP, at least at the time of the template deployment. This is because the template includes instructions to download certain required packages at the time of deployment (like the HA iApp), as well as do things like license the device. If you don't have this connectivity, you'll need to set this up manually (out of scope of this article). If using the template I have used, it actually needs internet access through both the mgmt interface and the ext interface. 2. The API call that is made at time of failover requires access to the internet from the BIG-IP. So this automatic update of the AWS route won't work without it. However, you could still make this API call yourself. You could write a script that updates the AWS route, along with any other things you need to do in case one of your AZ's becomes unreachable, for example, you may have application-level settings within your app to update in a real failover/disaster. So you could get by if your BIG-IP devices cannot reach the Internet. You'd need to manually deploy them and set them up (license, etc), and then script the failover scenario (updating AWS routes) yourself. Let me know if this helps!
hi @Michael OLeary
So let me get this straight...when i setup HA across AZ's for the purpose of load balancing internal services within a VPC. Failover wont work if my F5's dont have internet access?
Hi Sebastian. Failover of the devices will work without Internet access. The standby device will become active at the time of failover. However, if you'd like to have an AWS route automatically updated with an API call (which can be easily set up with the HA iApp), then you'll need the BIG-IP to be able to make an HTTPS connection to the (public) AWS API endpoint. The alternative would be to script this route update yourself for your failover scenario. Does that make sense?
HI Michael, one more question. So the F5 will failvoer.. (great .. thats normal)... Let me know if you can shed some light, how would i go about scripting this? would i do something on the f5? or would i use another server that requires access ? DM if you want 😄
Hi Sebastian, no problem. Likewise feel free to DM me. Yes, the F5 failover part is normal, and yes, updating any AWS routes automatically will require an API call. If you don't want to let the HA iApp set that up for you, you could script yourself. Consider this out of scope of this article, but you'll need to consider: a) you need to make a AWS API call to update the specific routes you have b) you can do this using a AWS cli command from a Windows or Linux box that has Internet access c) you should have some failure detection mechanism so that you can run this script when it is required. Obviously it's easier to let the HA iApp set this up, but if you have other non-F5 things to do at the time of failover, like update your application, then you could script this as part of your whole recovery activity. However, if you don't already use the AWS API for your own processes, I would recommend just using the HA iApp, especially to start with as you learn.
Cool, thanks .. last question 😄 can i use the API Gateway to handle this? (Does that make sense) ?
One more 😄 .. can i use aws cloud watch to watch the failure of the f5 and when a failure occurs use a lamda function to call the api to failover the routes ?
Hi Sebastian, I don't see why not, but I'd consider that beyond the recommendations of this article, because I haven't tested it. I like the idea though, getting those native services to do it instead of a different server. This is all to avoid allowing the BIG-IP appliance to access the internet and make the API call, right? Perhaps you could have the BIG-IP initiate that at failover, but use your own tools? If possible, could you research that and let us know how it works out? I'd be very interested in seeing that set up with tools that allowed you to make that AWS update without having the BIG-IP access the Internet via API call. I think for supportability though, the iApp is still the best bet.
Great, info.. I'm working on deploy BIG-IP in AWS with HA across AZ’s with EIP’s in my case. Do you have any details around how to get routing setup and configured between F5 and the backed servers in AWS?
Thanks,
Hi Wildzfor, if you are going to use EIPs, you still use a supported template and the HA iApp. Unless you SNAT all of your traffic to your backend servers, you are going to need a route in AWS so that return traffic from the backend servers routes out of the same BIG-IP device that it came in through. That's where the HA iApp can update it automatically for you. Follow the documentation and let us know how you go.
Hello,we deployed a VPC endpoint and enable private DNS for the endpoint indside Security VPC, with this configuration you can call to AWS API using public domains because inside the VPC the public urls are internally resolved to private addresses. We tested failover succefully.
Hello, I am doing the same scenario, almost the same architecture, I wanted to know about the iapp template, I kept asking myself for an elastic-ip to continue the display, what should I do or write there? , or how should that iapp be configured when the architecture has been defined the same as you have indicated?
Hi Omar, you should choose "No, do not configure high availability across Availability Zones" from the field above and then you will not enter any EIP. I know this sounds counter-intuitive, but that field is assuming you will use EIP's. In this architecture discussed, we are not using EIP's.
Also - at this point you should really be using the Cloud Failover Extension (CFE) which was released about a year ago, and is mentioned in an update to the article above. The CFE allows you to declare a failover configuration as a JSON template, and that iApp is no longer the way to do it going forward into the future.
Feel free to email me or reach out directly if you need help.
hello, michael, first thank you for your support, I have some questions:
1.- "should really be using the Cloud Failover Extension (CFE)" do you mean that I can no longer use the iapp ha? or will it stop working?
2.- regarding CFE, I am having a lot of trouble following the documentation https://clouddocs.f5.com/products/extensions/f5-cloud-failover/latest/userguide/quickstart.html I don't know if there is another article where Explain better, I am following the steps and when I run the api rest https: // {{host}} / mgmt / shared / cloud-failover / info I get another message from the one shown in the guide.
Now about the statements that are placed in the API, I don't know what should be put exactly in the json and where that should be put in the postman.
If it is possible you could indicate your email to make your queries is only the final part of a deployment that I am doing, only that I am missing, my email is jenciso@securesoftcorp.com, do you have any email?
Hi Omar
Mike.