Technical Articles
F5 SMEs share good practice.
cancel
Showing results for 
Search instead for 
Did you mean: 
Custom Alert Banner
MichaelOLeary
F5 Employee
F5 Employee

Background:

The CloudFormation templates that are provided and supported by F5 are an excellent resource for customers to deploy BIG-IP VE in AWS. Along with these templates, documentation guiding your F5 deployment in AWS is an excellent resource.

And of course, DevCentral articles are helpful. I recommend reading about HA topologies in AWS to start. I hope my article today can shed more light on an architecture that will suit a specific set of requirements: No Elastic IP's (EIP’s), High Availability (HA) across AZ’s, and multiple VPC’s.

Requirements behind this architecture choice:

I recently had a requirement to deploy BIG-IP appliances in AWS across AZ’s. I had read the official deployment guide, but I wasn’t clear on how to achieve failover without EIP’s. I was given 3 requirements:

  1. HA across AZ’s. In this architecture, we required a pair of BIG-IP devices in Active/Standby, where each device was in a different AZ. I needed to be able to fail over between devices.
  2. No EIP’s. This requirement existed because a 3rd party firewall was already exposed to the Internet with a public IP address. That firewall would forward inbound requests to the BIG-IP VE in AWS, which in turn proxied traffic to a pair of web servers. Therefore, there was no reason to associate an EIP (a public IP address) with the BIG-IP interface. In my demo below I have not exposed a public website through a 3rd party firewall, but to do so is a simple addition to this demo environment.
  3. Multiple VPC’s. This architecture had to allow for multiple VPC’s in AWS. There was already a “Security VPC” which contained firewalls, BIG-IP devices, and other devices, and I had to be able to use these devices for protecting applications that were across 1 or more disparate VPC’s.

Meeting the requirements:

HA across AZ’s

This is the easiest of the requirements to meet because F5 has already provided templates to deploy these in AWS. I personally used a 2-nic device, with a BYOL license, deployed to an existing VPC, so that meant my template was this one. After this deployment is complete, you should have a pair of devices that will sync their configuration.

At time of failover

The supported F5 templates will deploy with the Advanced HA iApp. It is important that you configure this iApp after you have completed your AWS deployments. The iApp uses IAM permissions deployed with the template to make API calls to AWS at the time of failover. The API calls will update the route tables that you specify within the iApp wizard. Because this iApp is installed on both devices, either device can update the route in your route tables to point to its own interface.


Update as of Dec 2019

This article was first written Feb 2019, and in Dec 2019 F5 released the Cloud Failover Extension (CFE), which is a cloud-agnostic, declarative way to configure failover in multiple public clouds. You can use the CFE instead of the Advanced HA iApp to achieve high availability between BIG-IP devices in cloud.


Update as of Apr 2020

Your API calls will typically be sent to the public Internet endpoints for AWS EC2 API calls. Optionally, you can use AWS VPC endpoints to keep your API calls out to AWS EC2 from traversing the public Internet. My colleague Arnulfo Hernandez has written an article explaining how to do this.

No EIP’s

Configure an “alien IP range”

I’m recycling another DevCentral solution here. You will need to choose an IP range for your VIP network that does not fall within the CIDR that is assigned to your VPC. Let’s call it an “alien range” because it “doesn’t belong” in your VPC and you couldn’t assign IP addresses from this range to your AWS ENI’s. Despite that, now create a route table within AWS that points this “alien range” to your Active BIG-IP device’s ENI (if you’re using a 2+ nic device, point it to the correct data plane NIC, not the Mgmt. interface). Don’t forget to associate the route table with specific subnets, per your design. Alternatively, you could add this route to the default VPC route table.

Create a VIP on your active device

Now create a VIP on your active device and configure the IP address as an IP within your alien range. Ensure the config replicates to your standby device. Ensure that source/destination checking is disabled on the ENI’s that your AWS routes are pointing to (on both Standby and Active devices). You should now have a VIP that you can target from other hosts in your VPC, provided that the route you created above is applied to the traffic destined to the VIP.

Multiple VPC’s

For extra credit, we’ll set up a Transit Gateway. This will allow other VPCs to route traffic to this “alien range” also, provided that the appropriate routes exist in the remote VPC’s, and also are applied to the appropriate Transit Gateway Route Table. Again, I’m recycling ideas that have already been laid out in other DevCentral posts.

I won’t re-hash how to set up a transit gateway in AWS, because you can follow the linked post above. Sufficed to say, this is what you will need to set up if you want to route between multiple VPC’s using a transit gateway:

  1. 2 or more VPC’s
  2. A transit gateway in AWS
  3. Multiple transit gateway attachments that attach the transit gateway and each VPC you would like to route between. You will need one attachment per VPC.
  4. A transit gateway route table that is associated with each attachment.

 

I will point out that you need to add a route for your “alien range” in your transit gateway route table, and in your remote VPC’s. That way, hosts in your remote VPC’s will forward traffic destined to your alien range (VIP network) to the transit gateway, and the transit gateway will forward it to your VPC, and the route you created in Step A will forward that traffic to your active BIG-IP device.

Completed design:

After the above configuration, you should have an environment that looks like the diagram below:

0151T000003d7JkQAI.JPG

Tips

Internet access for deployments: When you deploy your BIG-IP devices, they will need Internet access to pull down some resources, including the iApp. So if you are deploying devices into your existing VPC, make sure you have a reachable Internet Gateway in AWS so that the devices have Internet access through both their management interface, and their data plane interface(s).

Internet access for failover: Remember that an API call to AWS will still use an outbound request to the Internet. Make sure you allow the BIG-IP devices to make outbound connections over HTTPS to the Internet. If this is not available, you will find that your route tables are not updated at time of failover. (If you have a hard requirement that your devices should not have outbound access to internet, you can follow Arnulfo's guide linked to above and use VPC endpoints to keep this traffic on your local VPC)

iApp logs: you can enable this in the iApp settings. I have used the logs (in /var/ltm/log) to troubleshoot issues that I have created myself. That’s how I learned not to accidentally cut off Internet access for your devices!

Don’t forget about return routes if SNAT is disabled: Just like on-prem environments, if you disable SNAT, the pool member will need to route return traffic back to the BIG-IP device. You will commonly set up a default route (0.0.0.0/0) in AWS, point it at the ENI of the active BIG-IP device, and associate this route table with the subnet containing the pool members. If the pool members are in a remote VPC, you will need to create this route on the transit gw route table also.

Don’t accidentally cut off internet access: When you configure the default route of 0.0.0.0/0 to point to eth1 of the BIG-IP device, don’t apply this route to every subnet in your Security VPC. It may be easy to do so accidentally, but remember that it could cause the API calls to update route tables to fail when the Standby device becomes Active.

Don’t forget to disable source/dest check on your ENI’s. This is configured by the template, but if you have other devices that require it, remember to check this setting.

Comments

Awesome! Quick question, can i route that "Alien range" across different VPCS?

 

MichaelOLeary
F5 Employee
F5 Employee

Hi Sebastian, yes - but you have to use a Transit Gateway to route between your VPCs. Transit Gateways were released in late 2018, and they will allow you to route between multiple VPCs. Prior to this, you needed to manage VPC peering relationships individually, and these peering relationships did not allow you to route a non-VPC IP range. (The other option prior to the release of Transit Gateways was to build IPSec VPN connections yourself between VPCs and I have not covered that here).

 

Thanks Michael.

 

Yes i didnt want to go down the IPsec vpn route.

 

I need to test this out ASAP 😄

 

rafaelbn
Cirrostratus
Cirrostratus

Hello Michael! Thanks for the article. Quick question. Since bigip-1 and bigip-2 are on different AZs, and that means different subnets, how did you solve the default route after you clustered? Not the management routes, the TMOS routes (because they sync these stuff)... Cheers, Rafael!

 

MichaelOLeary
F5 Employee
F5 Employee

Hi Rafael, I deployed this template from F5's supported sample templates. I recommend doing the same. The template deployment configures default routes into a LOCAL_ONLY partition on each device, so that this default route for a device does not sync to the paired device.

 

amintej
Cirrus
Cirrus

Hello Michael, thank you for the article, it is really interesting and very useful for us because our desgin don't include EIPs. In case, F5 can't connect to the Internet, do you have workaround for this situation ? Br

 

MichaelOLeary
F5 Employee
F5 Employee

Hi Amintej, glad to hear it's useful. 2 things I can think to say in response: 1. If using the template I did, you will need internet connectivity from the BIG-IP, at least at the time of the template deployment. This is because the template includes instructions to download certain required packages at the time of deployment (like the HA iApp), as well as do things like license the device. If you don't have this connectivity, you'll need to set this up manually (out of scope of this article). If using the template I have used, it actually needs internet access through both the mgmt interface and the ext interface. 2. The API call that is made at time of failover requires access to the internet from the BIG-IP. So this automatic update of the AWS route won't work without it. However, you could still make this API call yourself. You could write a script that updates the AWS route, along with any other things you need to do in case one of your AZ's becomes unreachable, for example, you may have application-level settings within your app to update in a real failover/disaster. So you could get by if your BIG-IP devices cannot reach the Internet. You'd need to manually deploy them and set them up (license, etc), and then script the failover scenario (updating AWS routes) yourself. Let me know if this helps!

 

hi @Michael OLeary

 

So let me get this straight...when i setup HA across AZ's for the purpose of load balancing internal services within a VPC. Failover wont work if my F5's dont have internet access?

 

MichaelOLeary
F5 Employee
F5 Employee

Hi Sebastian. Failover of the devices will work without Internet access. The standby device will become active at the time of failover. However, if you'd like to have an AWS route automatically updated with an API call (which can be easily set up with the HA iApp), then you'll need the BIG-IP to be able to make an HTTPS connection to the (public) AWS API endpoint. The alternative would be to script this route update yourself for your failover scenario. Does that make sense?

 

Thanks Michael thats what i thought. Makes sense.

 

HI Michael, one more question. So the F5 will failvoer.. (great .. thats normal)... Let me know if you can shed some light, how would i go about scripting this? would i do something on the f5? or would i use another server that requires access ? DM if you want 😄

 

MichaelOLeary
F5 Employee
F5 Employee

Hi Sebastian, no problem. Likewise feel free to DM me. Yes, the F5 failover part is normal, and yes, updating any AWS routes automatically will require an API call. If you don't want to let the HA iApp set that up for you, you could script yourself. Consider this out of scope of this article, but you'll need to consider: a) you need to make a AWS API call to update the specific routes you have b) you can do this using a AWS cli command from a Windows or Linux box that has Internet access c) you should have some failure detection mechanism so that you can run this script when it is required. Obviously it's easier to let the HA iApp set this up, but if you have other non-F5 things to do at the time of failover, like update your application, then you could script this as part of your whole recovery activity. However, if you don't already use the AWS API for your own processes, I would recommend just using the HA iApp, especially to start with as you learn.

 

Cool, thanks .. last question 😄 can i use the API Gateway to handle this? (Does that make sense) ?

 

One more 😄 .. can i use aws cloud watch to watch the failure of the f5 and when a failure occurs use a lamda function to call the api to failover the routes ?

 

MichaelOLeary
F5 Employee
F5 Employee

Hi Sebastian, I don't see why not, but I'd consider that beyond the recommendations of this article, because I haven't tested it. I like the idea though, getting those native services to do it instead of a different server. This is all to avoid allowing the BIG-IP appliance to access the internet and make the API call, right? Perhaps you could have the BIG-IP initiate that at failover, but use your own tools? If possible, could you research that and let us know how it works out? I'd be very interested in seeing that set up with tools that allowed you to make that AWS update without having the BIG-IP access the Internet via API call. I think for supportability though, the iApp is still the best bet.

 

Wildzfor_386457
Nimbostratus
Nimbostratus

Great, info.. I'm working on deploy BIG-IP in AWS with HA across AZ’s with EIP’s in my case. Do you have any details around how to get routing setup and configured between F5 and the backed servers in AWS?

 

Thanks,

 

MichaelOLeary
F5 Employee
F5 Employee

Hi Wildzfor, if you are going to use EIPs, you still use a supported template and the HA iApp. Unless you SNAT all of your traffic to your backend servers, you are going to need a route in AWS so that return traffic from the backend servers routes out of the same BIG-IP device that it came in through. That's where the HA iApp can update it automatically for you. Follow the documentation and let us know how you go.

 

amintej
Cirrus
Cirrus

Hello,we deployed a VPC endpoint and enable private DNS for the endpoint indside Security VPC, with this configuration you can call to AWS API using public domains because inside the VPC the public urls are internally resolved to private addresses. We tested failover succefully.

 

MichaelOLeary
F5 Employee
F5 Employee

That is so interesting amintej, thanks for sharing that!

 

Jeff_Giroux
Cirrus
Cirrus

This is a great write up. Thx!

omar_padilla
Altocumulus
Altocumulus

Hello, I am doing the same scenario, almost the same architecture, I wanted to know about the iapp template, I kept asking myself for an elastic-ip to continue the display, what should I do or write there? , or how should that iapp be configured when the architecture has been defined the same as you have indicated?

0691T000008tn2wQAA.png

MichaelOLeary
F5 Employee
F5 Employee

Hi Omar, you should choose "No, do not configure high availability across Availability Zones" from the field above and then you will not enter any EIP. I know this sounds counter-intuitive, but that field is assuming you will use EIP's. In this architecture discussed, we are not using EIP's.

 

Also - at this point you should really be using the Cloud Failover Extension (CFE) which was released about a year ago, and is mentioned in an update to the article above. The CFE allows you to declare a failover configuration as a JSON template, and that iApp is no longer the way to do it going forward into the future.

 

Feel free to email me or reach out directly if you need help.

omar_padilla
Altocumulus
Altocumulus

hello, michael, first thank you for your support, I have some questions:

1.- "should really be using the Cloud Failover Extension (CFE)" do you mean that I can no longer use the iapp ha? or will it stop working?

2.- regarding CFE, I am having a lot of trouble following the documentation https://clouddocs.f5.com/products/extensions/f5-cloud-failover/latest/userguide/quickstart.html I don't know if there is another article where Explain better, I am following the steps and when I run the api rest https: // {{host}} / mgmt / shared / cloud-failover / info I get another message from the one shown in the guide.

 

Now about the statements that are placed in the API, I don't know what should be put exactly in the json and where that should be put in the postman.

 

0691T000008tpEGQAY.png

If it is possible you could indicate your email to make your queries is only the final part of a deployment that I am doing, only that I am missing, my email is jenciso@securesoftcorp.com, do you have any email?

MichaelOLeary
F5 Employee
F5 Employee

Hi Omar

  1. The iApp won't stop working, it is just being deprecated. I just downloaded the latest iApp templates from downloads.f5.com and it included the AWS HA iApp, with a helpful ReadMe txt file that included this line: "The planned EoSD/EoTS/EoL date for this iApp template is 12/31/20." Also, you can check out this KB article for details about iApp deprecation: https://support.f5.com/csp/article/K13422.
  2. Good job on starting with the CFE! I'll gladly help you out over email and we can set up a call if needed. I'll shoot you an email now.

Mike.

Qiang_LI
Legacy Employee
Legacy Employee

Hi,Michael, I'm doing poc just like what your article's description.Did you have some more detail guide or setup manual  which I can follow to setup my POC more easily?

  Thanks!

Qiang 

boneyard
MVP
MVP

I know it is a while ago, but big kudos on keeping an article updated with more recent developments. Cloud is moving fast.

cnelvitigala
Nimbostratus
Nimbostratus

Awesome  Articale  with great info  ..!!!

Version history
Last update:
‎21-Feb-2019 05:00
Updated by:
Contributors