F5 BIG-IP deployment with Red Hat OpenShift - keeping client IP addresses and egress flows

Introduction

In this article it will be covered how to control the egress traffic in the Red Hat OpenShift cluster making use of the AdminPolicyBasedExternalRoute resource type introduced with OpenShift 4.14.

This functionality can be used for several use cases:

• Keeping the source IP of the ingress clients by setting the BIG-IP as default gateway for the namespaces which the BIG-IP provides load balancing.
• Providing highly scalable SNAT for egress flows to outside the cluster. This includes per namespace SNATs.
• Providing security functionalities for egress flows to outside the cluster, such as firewall, IDS/IPS, malware protection, reporting and analytics, data loss prevention, URL filtering and integration with third party security solutions by doing SSL/TLS inspection.

Keeping the source IP of ingress clients

By default OpenShift sends the traffic to the default gateway of the network where the OpenShift cluster is placed. In the case of ingress traffic to the applications, in order to make the traffic symmetric, making the return traffic go through the load balancer, it has been required to use SNAT in the load balancer.

With the introduction AdminPolicyBasedExternalRoute this is no longer required, allowing the use the source IP to the workload PODs (in the case of 1-tier deployment) or to the in-cluster ingress controller (in the case of 2-tier deployments). The overall traffic flows in the platform would be as shown next:

Keeping the source IP of the clients

Using the feature is straight forward, we only have to specify the namespaces for which we want the BIG-IP to be the gateway and the floating IP of the BIG-IPs facing the OpenShift cluster. An example is shown next: 

apiVersion: k8s.ovn.org/v1
kind: AdminPolicyBasedExternalRoute
metadata:
  name: meg-policy-common
spec:
  from:
    namespaceSelector:
      matchLabels:
        meg: common
  nextHops:
    static:
    - ip: "10.1.10.100"

The labels can be freely chosen, in this case it is chosen "meg" as label (meg standing multi-egress gateway) and the value "common." In the next sections we will see further possibilities. 

Providing highly scalable SNAT for egress flows

One egress SNAT for all namespaces handled by the BIG-IP

Another use case of this functionality is to delegate the SNAT function into BIG-IP, this allows for highly scalability SNAT for egress flows, this is traffic initiated from the cluster to the outside which is not load balanced. The scalability benefits are two fold:

• BIG-IP has a carrier-grade NAT which allows high number of concurrent flows and great configuration flexibility.
• By doing the SNAT function in BIG-IP, it is possible to SNAT the traffic of all nodes to a single IP instead of assigning one SNAT address per node. This allows managing the traffic through firewalls easier because filters don´t need to be changed when nodes are added or removed.

Besides of the above, recall that this configuration in the BIG-IP can be done, if needed, in a per namespace basis, as shown next. 

Using SNAT in the BIG-IP for egress traffic.

This setup requires a virtual server which typically is not modified and doesn´t need to be created with CIS (it could be created with CIS with an AS3 ConfigMap if desired). The configuration is fairly simple:

Configuration of an egress virtual server in the BIG-IP

From this configuration it is worth highlighting:

• Specifying the VS type as Forwarding (IP)
• Restricting the source IPs with the range of the PODs
• Specifying the desired SNAT pool

Multiple egress SNATs for namespaces handled by the BIG-IP

It might be desired to have different sets of SNATs depending on the namespaces, this can be used by external firewalls to differentiate between applications. This can be accomplished by means of having multiple virtual servers as shown next:

 

Using different SNAT pools for different namespaces by means of having multiple egress virtual servers

In OpenShift, there will be a separate AdminPolicyBasedExternalRoute manifest for each set of namespaces where we want to apply separate SNAT pools. In the figure above, these manifests are the "meg-policy-common" shown above and a new "meg-policy-deciated" shown next:

apiVersion: k8s.ovn.org/v1
kind: AdminPolicyBasedExternalRoute
metadata:
name: meg-policy-dedicated
spec:
from:
  namespaceSelector:
    matchLabels:
      meg: dedicated
nextHops:
  static:
    - ip: "10.1.10.200"

Each AdminPolicyBasedExternalRoute points to a different floating-IP in the BIG-IP. But the BIG-IP cannot differentiate in the forwarding VS in which floating-IP the traffic was received because all the IPs in a given VLAN use the same MAC address.

Because of the latter, each virtual server needs to be listening in a different VLAN so the BIG-IP can differentiate from which namespace the traffic came from. As you might have noted, typically the OpenShift nodes only sit in a single VLAN. To solve this, dummy VLANs in the BIG-IP will be created connecting to the same OpenShift network but with different interfaces. Furthermore, in order to have the same subnet in these VLANs it will be needed to assign each VLAN to a different route domain.

Overall, the setup will be as follows:

Configuration for multiple egress

In practice the configuration is very similar to the case where there is a single egress virtual server:

Configuration of an egress virtual server in the BIG-IP allowing additional virtual servers by means of using route domains

Note that from the previous configuration we only had to add the route domain ID (in this case %10) to the IP addresses. The routing domains created (10 and 20) are shown next:

The full configuration of one of these route domains is shown next as example. The only required parameter is to have as parent the route domain 0 because it will be used to reach the external VLAN.

The self-IPs of the BIG-IP facing the OpenShift clusters have to be in the corresponding route domains:

Note that it is required to have separate non-floating IP addresses as well.

Providing security functionalities for egress flows

Once the BIG-IP is the gateway of the applications, it can be used to add many functionalities. This is outlined in the next figure:

 

These functionalities are delivered with the SSL Ochestrator and Secure Web Gateway Services add-on modules. 

Furthermore, these modules allow the inclusion of additional third-party security integrations, like Cisco Firepower, Cisco WSA, Symantec DLP, RSA Netwitness, FireEye NX and PaloAlto NG Firewall.

Conclusion and final remarks

The AdminPolicyBasedExternalRoute resource type introduced with OpenShift 4.14 opens many possibilities of getting more visibility into the cluster, higher scalability and many security functionalities available in your existing BIG-IP platform. I hope this article has been an eye opener for the possibilities of the BIG-IP platform with OpenShift. I would love to hear if you have any specific requirements for the use cases mentioned in the article.