SSLO in public cloud: Azure inbound L3 use case
Summary
I have a customer who uses BIG-IP SSL Orchestrator (SSLo) on premises for inbound traffic, and they wanted to replicate this architecture in several Azure regions. This article explains how to achieve this, and is specifically focused on an Inbound L3 use case in Azure.
An overview of SSL Orchestrator functionality
Introduction
When I first learned about SSLo, I was struck because the architecture made so much sense. I was a customer at the time. We had two different firewalls that were daisy-chained in the path of all inbound/outbound Internet traffic, performing TLS decryption on each. This caused typical problems and an informal team narrative of "don't fix an issue unless you want to own it forever!" Here's an overview of the problems we had.
Inbound TLS decryption problems
- TLS certificates for our websites were stored in 3-4 places (2x firewalls, 1x BIG-IP, and often the app server). This meant:
- Certificate expiration often caused an outage and inevitable finger-pointing
- Certificate inventory/tracking was difficult.
- Latency for inbound traffic (ie, the customer experience) was negatively affected and ignored.
Outbound TLS decryption problems
- Unsurprisingly, outbound web traffic was an operational headache to decrypt with our man-in-the-middle approach using two separate firewalls. Why?
- Each device had it's own database of website categories. If we wanted to not decrypt healthcare website traffic, each device had different ideas about what constituted healthcare.
- Exceptions had to be whitelisted in multiple places by different teams (takes time, invites inconsistency, breaks a lot of automation, etc).
- Operations (upkeep of whitelists, troubleshooting, Root and Intermediate CA cert upkeep, etc) often led to more finger-pointing.
- Users became unhappy and found dangerous workarounds (think home VPN's or iPhone tethering) to get around IT Security.
Fortunately SSLo offers a great alternative that not only solves these problems but offers more solutions too. Those solutions hit home when I was considering my recent customer who uses SSLo for inbound traffic on-premises and wanted to do the same in Azure. (I will write a future post about outbound TLS decryption in Azure. For now, this article focuses on inbound traffic only).
SSLo for inbound traffic decryption in Azure
My customer wanted to replicate their on-premises SSLo architecture with the same in Azure, for good reason. I'll list their requirements:
- They wanted to decrypt all inbound HTTPS traffic so that they could run it through a WAF "in the clear". Since WAF's don't handle decryption very well, decrypting at the WAF would require purchase of a lot more WAF devices. Ideally decryption is done before a WAF.
- They wanted to send only some traffic through the WAF. Somewhat obviously, non-HTTP traffic can bypass their WAF since it's not configured to secure traffic of this type. So there's no reason to have the WAF inline for all inbound traffic (requiring additional WAF devices simply to route this traffic volume).
- They wanted the ability to select a different WAF in future, or selectively whitelist certain source IP's from WAF inspection, or add an additional inspection device in future, or - you get the idea - they wanted to be able to orchestrate inbound TLS traffic between multiple devices.
My Azure lab architecture
Here's a lab I put together to show a working SSLo architecture in Azure.
Note: this customer also wants to use the Azure Gateway Load Balancer and have a separate pair of Palo Alto devices doing nothing but Layer 3/4 edge firewalling. So I have built that out for them too.
Now that we've seen an overview of their architecture, I will run through some of the finer points of this lab for anyone considering this architecture.
Azure Gateway Load Balancer for edge firewalls
I've written about Azure Gateway Load Balancer (GWLB) in a previous article. I discussed how to put BIG-IP behind Azure GWLB, but in this scenario the customer wants to use Palo Alto devices as an "edge" firewall. They aren't decrypting traffic here, just performing L3/L4 inspection. Azure's GWLB allows them to run multiple standalone firewalls without worrying about src/dest NAT'ing and other network challenges. I used this guide to set up Palo Alto devices behind Azure's GWLB.
BIG-IP deployed in High Availability pair
BIG-IP is deployed as Active/Standby pair behind a standard public Azure Load Balancer (LB). This LB has a front end IP config that referenes the GWLB in another VNET, but then passes traffic without performing Destination NAT to BIG-IP. This is done by enabling "floating IP" on the Azure LB rule. The BIG-IP has a VIP that is listening on the same public IP that is configured as the Azure LB front end IP. This way, BIG-IP sees true source and destination IP.
SSLo is licensed and configured on BIG-IP for an Inbound L3 architecture, which is selected in the Guided Configuration wizard. The wizard allows me to apply TLS decryption at the VIP, pass traffic unencrypted to a pool of inspection devices, and to expect that traffic to be returned to the BIG-IP on another interface. After that inspection, traffic is then sent by BIG-IP to pool members. SNAT and reencryption are possible, but in this case I'm only SNAT'ing traffic before sending unencrypted to web servers.
If you're looking at my lab diagram, you should know that 10.0.6.10 and 10.0.7.10 are the floating Self IP's I have on the BIG-IP's, in the subnets "sslo-in" and "sslo-out" respectively. These are referenced in the UDR's required for routing. Upon failover of this Active/Standby pair, the Cloud Failover Extension (CFE) ensures these IP addresses move between the 2 Azure VM's.
Palo Alto for WAF inspection
Many customers use Palo Alto as a WAF inspection device. This is a different set of devices from the "edge" devices that are behind GWLB. These devices expect to receive traffic unencrypted on HTTP port 80. They are set up in a basic manner: they have 1 NIC and Azure's default GW for the subnet that this NIC is in. However, I have also done 2 interesting things with these Palo Alto devices:
- I've enabled an Interface Management Profile that allows the relevant self IP of the BIG-IP to hit the management console on HTTP on the dataplane NIC. This is because BIG-IP will send a health check to it's inspection devices.
- I've added a /32 IP address to the data plane NIC, and this IP address matches the frontend IP config of the Azure load balancer in front of the Palo Alto WAF's. This is because I have a single LB rule on this Azure LB, where all traffic, all TCP ports, is forwarded with "floating IP" (ie, no DNAT) checked on the LB rule. So, the health check I'm talking about in the point above will reach the PA with the destination IP address of the Azure LB.
If we reference my lab diagram above, 10.0.7.6 is the IP address of the frontend IP configuration in the Internal Azure Load Balancer. The Palo Alto appliances (I have deployed 2 of them) have a single NIC each, and their IP addresses are 10.0.3.4/24 and 10.0.3.5/24. But, they both have an additional IP address of 10.0.7.6/32 assigned to the same NIC only for the purpose of health checks from F5 BIG-IP.
Why would the PA firewalls need the additional /32 address? Because I have the Internal Azure Load Balancer configured with a single LB rule: allow HA ports (all ports), and enable Floating IP (don't perform Destination NAT). The BIG-IP believes it has a single inspection device at 10.0.7.6, and sends health checks on HTTP to this address. The Azure LB forwards these health checks, without NAT'ing, to their backend pool members.
Conclusion
My main goal for this article is to demonstrate a working architecture of SSLO that allows for High Availability in an inbound, L3 scenario in Azure. Please, message me or leave a comment if you want more info, or reach out to your account team. And thanks for reading!
