F5 Cloud Failover Extension (CFE), private endpoints, and custom DNS

Summary

When using the F5 Cloud Failover Extension (CFE) for API-based failover in public cloud, some customers block API calls out to the public Internet. Private endpoints solve for this constraint, but rely on DNS. If a customer is using custom DNS servers, a workaround may be required.

F5 CFE

The CFE is a package that runs on BIG-IP to perform failover in public cloud. It makes a series of API calls to the cloud provider to move IP addresses between interfaces, update EIP associations, or update route tables. It is supported in Azure, AWS, and Google Cloud (GCP).

 

Typical case of CFE: API calls to update cloud configuration at failover.

Customer problem statement

My customer was successfully using CFE to move IP addresses between interfaces in Azure. They had followed documentation and created storage account that was publicly available, but only accessible by the BIG-IP VM’s in Azure.

They decided to create a private endpoint for their storage account in Azure and block Internet access to their storage account. CFE stopped working and reported “403 Forbidden” errors in logs. 

The issue was their use of custom DNS settings. As we’ll discuss, a simple workaround fixed their problem. 

CFE in isolated environments

When CFE makes API calls to a cloud provider, the destination of these calls is a public API endpoint. For example:

• In Azure, storage API calls may be sent to https://xxx.blob.core.windows.net
• In AWS, storage might be accessed at https://s3.us-east-2.amazonaws.com
• In GCP, storage calls might be made to https://storage.googleapis.com

Since the BIG-IP is already running in the cloud provider, it’s possible to access these services without traversing the Internet. Private endpoints allow a VM to access services within these cloud providers directly, and they can be set up in Azure, AWS, and GCP.

Documentation exists to help F5 customers configure private endpoints in Azure and GCP. A DevCentral article also walks through this for AWS users.

Azure, Custom DNS, and CFE in isolated environments

My recent customer was running CFE in Azure, but this should apply to other cloud providers also.

Private endpoints work by using DNS. Once a private endpoint is configured, DNS entries are added for the networks that will use the private endpoint.

For example, if I have an Azure storage account called olearystorageacct, blobs in this storage account will be reachable at olearystorageacct.blob.core.windows.net. This might resolve to a public IP of 1.1.1.1. If I disable public Internet access to my Storage Account and create a private endpoint for it, VM’s in my VNET will now resolve this name to a private IP address from their own VNET, such as 10.1.1.5 in the diagram below.

 

An example diagram of a private endpoint for Azure storage account access.

 

However, this assumes VM’s are using the default DNS settings in the Azure VNET. When using custom DNS, VM’s will resolve DNS queries against configured servers. In the diagram below, 10.0.0.100 and 101 are IaaS DNS servers.

 

Example flow of DNS queries when custom DNS servers are used in VNET settings.

 

Working with custom DNS settings and private endpoints

Custom DNS represents a problem here. The IaaS DNS servers will forward DNS queries out to Internet-based DNS servers for domains like blob.core.windows.net. Despite the existence of a private endpoint, VM’s using custom DNS may still resolve this record to a public IP address. How can we overcome this?

I quickly found others with the same problem, and read related Microsoft documentation that got my mental wheels turning.

I came up with this list of options for my customer, ordered from easiest to hardest in my opinion:

1. Set a hosts record on your BIG-IP device, so that olearystorageacct.blob.core.windows.net is manually set to the IP of the private endpoint, e.g. 10.1.1.5
2. Set the DNS server of BIG-IP to Azure’s DNS server at 168.63.129.16. (However, this means every DNS request from this VM will use Azure’s DNS and not custom DNS servers.)
3. Create a forwarding rule on the custom DNS server to forward requests for *.blob.core.windows.net to 168.63.129.16.
4. Create an A record on the custom DNS server: olearystorageacct.blob.core.windows.net could be manually set to 10.1.1.5
5. Lastly, a customer could always move away from custom DNS, and revert to default Azure settings for DNS. They can still use private DNS zones and Azure’s Private DNS Resolver. This is a larger architectural change of course, not a simple workaround.

My customer chose Option 2. CFE started working again! Their BIG-IP VM is using Azure’s DNS server and accessing the storage account via private endpoint, but the custom DNS servers are in place for all other VM’s following the VNET settings.

Conclusion

In retrospect, this was a simple solution. When using both private endpoints and custom DNS together, plan for how DNS resolution will work for given services. I hope this article helps if you find yourself in the same situation!