F5 BIG-IP deployment with OpenShift - per-application 2-tier deployments

Introduction

This article dives deeper in 2-tier deployments introduced in the previous article F5 BIG-IP deployment with OpenShift - platform and networking options yet we will start comparing a common Kubernetes/OpenShift architecture with a more capable and flexible architecture using BIG-IP. Although this type of deployment can be done with the BIG-IP in Layer 4 (L4) mode this article will focus in using the BIG-IP in Layer 7 (L7) mode which allows for more functionality and flexibility.

A regular Kubernetes/OpenShift deployment is as follows:

Typically, the External LB doesn´t add much added value and just spreads the connections across the ingress controller instances without any service intelligence.

This article emphasizes on the added value of F5 BIG-IP with Container Ingress Services (CIS) by operating at L7. Additionally, it is described how CIS is ingress agnostic and can be used at is fullest regardless of the ingress controller, API manager, service mesh used, or even a combination of these!

This 2-tier architecture can be seen next, where it is not only shown that traffic is sent directly to the ingress controllers without the NodePort indirection. In the figure, it can also be seen compared to a 1-tier architecture:

 

ClusterIP and NodePort

When using a 2-tier deployment, a key aspect that needs to be decided is how it is going to be sent the traffic to the ingress controllers.

NodePort

With other External LB solutions (ELB), the common way to send traffic to the ingress controllers in Kubernetes is by exposing these as NodePort.

This option always works because the ELB doesn't need to deal with the CNI. In this case, when the ELB sends traffic to a node, the NodePort construct by default load balances the traffic to any node in the cluster where the ingress controller is. This means that there would be 3 hops before reaching the workload POD. These hops are: the ELB, the ingress controller, the NodePort, and finally the workload POD.

This NodePort load balancing to other nodes can be removed by changing the externalTrafficPolicy from Cluster to Local in the Service definition. Thus reducing the number of hops from 3 to 2.

ClusterIP

Alternatively, the BIG-IP can send the traffic directly to the POD addresses, aka ClusterIP mode, without any translation or Kubernetes construct in between the BIG-IP and the ingress controller. This is specified in CIS with the --pool-member-type=cluster option and requires support of the CNI.

Using OpenShiftSDN and ClusterIP is discouraged because this CNI is being deprecated and will not be covered in this article.

When using OVNKubernetes, the BIG-IP sends the traffic without any tunnelling. At this moment, this requires that the clusters are in the same subnet as the BIG-IP.

Which option to choose?

• If the OpenShift cluster is not in the same subnet as the BIG-IP or if the infrastructure doesn´t allow POD addresses traverse the segment (ie: because of a L2 firewall/micro-segmentation), then NodePort would need to be used.
• I would discourage the use of NodePort with externalTrafficPolicy: Cluster because it doesn´t allow to do adequate health check monitoring because each probe will be sent to a different ingress controller. Moreover, persistence and troubleshooting is more tricky too because of the additional hop that NodePort creates.
  If NodePort needs to be used, then externalTrafficPolicy: Local referencing the ingress controllers is recommended.
• If there is no problem in routing the traffic to the OpenShift nodes using the POD addresses, I would encourage using ClusterIP because it is a transparent method, where the application traffic is easier to track: there are no additional hops or address translations.

OpenShift´s bundled ingress/router is HA-proxy; In on-prem OpenShift deployments, the default router instance is created using hostNetwork access. HostNetwork can also be used with CIS --pool-member-type=cluster mode. The IP addresses of the pool members will be the node addresses, but there will not be any indirection to the POD.

The examples provided in the git repository mentioned in this article use ClusterIP but can be easily modified to NodePort if necessary.

2-tier per-service load balancing and health checking

This type of deployment requires that the external load balancer, in this case the F5 BIG-IP, can take actions based on the L7 route path. This includes health-checking the application and performing TLS/SSL termination.

From a Kubernetes/OpenShift point of view, the most remarkable is that it is needed to expose the L7 routes twice. That is, for a given L7 route there will be a manifest for the ingress controller and another L7 route manifest for CIS. This is outlined in the next figure:

In the above figure, there is only one ingress controller instance with several replicas serving the same L7 routes. It is not relevant which ingress controller it is. This ingress controller can have defined its L7 routes with any resource type. This is transparent from the point of view of CIS. CIS only requires that the ingress controller can be referenced with a selector from a Service. This service is shown in brown.

There will be a Service manifest for each L7 route for CIS. Even if all these services point to the same ingress controller instances. This allows you to:

• Monitor each L7 route´s backends individually.
• Have a clean NetOps/DevOps separation where DevOps can manipulate the ingress controller configurations freely and NetOps can control the publishing of these in the ELB.
• Ultimately make load-balancing decisions in a per-application basis

The L7 routes for CIS can be defined either as VirtualServer, Route (as NextGen Route) or as Ingress resource types. Using the Ingress resource type is not recommended because of its limited functionalities compared with the other types.

Overall, there will be a 1:1:1 mapping between the L7 routes defined for CIS, the Service manifests referenced by CIS´ L7 routes, and the L7 routes defined for the ingress controller. This is shown conceptually in the next figure.

With respect to load balancing algorithm, given that the number of ingress controllers in a two-tier deployment is typically small it is sensible to take into account the load of each ingress controller by using an algorithm such as least-connections, least-sessions (if using persistence) or fastest (application response time) which should send more traffic to the ingress controller which is performing better or in other words, would avoid sending more traffic to an ingress controller that is not performing well.

Unify all ingresses in a single VIP

Thanks to CIS being ingress controller agnostic, it is also possible to combine L7 routes from different ingress controllers, these can be a mix of one or more ingress controllers, including service mesh ingress or API managers. This brings the flexibility of exposing L7 routes from these sources in a single VIP, as shown in the next figure.

From the picture above, it can be seen how in this arrangement the ingress controllers are not limited to a single L7 route. Also note that as mentioned previously, it is needed to have a 1:1 mapping between the L7 routes defined in the BIG-IP (1st tier) and in the in-cluster ingress element (2nd tier).

By default, these paired L7 routes are expected to match their URL. If necessary, it is possible to do URL translation by using the hostRewrite and rewrite attributes in the VirtualServer CR.

OpenShift´s documentation mentions several use cases for creating additional OpenShift router (HA-proxy) instances, referred to as route sharding. These are:

• Balance Ingress Controllers, or routers, with several routes to speed up responses to changes.
• Allocate certain routes to have different reliability guarantees than other routes.
• Allow certain Ingress Controllers to have different policies defined.
• Allow only specific routes to use additional features.
• Expose different routes on different addresses so that internal and external users can see different routes, for example.
• Transfer traffic from one version of an application to another during a blue-green deployment.

An additional use case is to isolate the control-plane Routes of the cluster and the application Routes in independent HA-proxy instances. This way it is not only it can be guaranteed different levels of reliability but also guaranteeing resource allocation for the cluster´s own traffic.

In the section "Using additional Router instances" section below, it is shown how to do these configurations.

Monitoring in a 2-tier deployment

For each L7 route, regardless of the resource type used (Ingress, Route or VirtualServer), these share the same schema for monitoring.

Next are shown the relevant parameters, where send is the key parameter that specifies L7 route:

monitor:
  type: http
  send: "GET / HTTP/1.1\r\nHost: www.twotier.com\r\nConnection: close\r\n\r\n"
  recv: "^HTTP/1.1 200"

In many cases, https will be used instead. HA-proxy requires that the health monitor uses TLS 1.2. By default, the https monitor defaults to TLS 1.0, hence we will need to specify one of the BIG-IP default TLS profiles as shown next:

monitor:
  type: https
  send: "GET / HTTP/1.1\r\nHost: www.twotier.com\r\nConnection: close\r\n\r\n"
  recv: "^HTTP/1.1 200"
  sslProfile: /Common/serverssl

When using NGINX or Istio ingress gateway, using TLS SNI (Server Name Indication) is requierd. In cases where SNI is required, the following needs to be done:

• One time, in TMM monitoring needs to be enabled as indicated in https://my.f5.com/manage/s/article/K85210713
• For each FQDN it is needed to pre-create a server-side TLS profile in the partition /Common specifying the server name used as TLS SNI.

The resulting monitor section will typically look as follows:

monitor:
  type: https
  send: "GET / HTTP/1.1\r\nHost: www.twotier.com\r\nConnection: close\r\n\r\n"
  recv: "^HTTP/1.1 200"
  sslProfile: /Common/serverssl-www.twotier.com

Note that this TLS profile is only needed for the monitoring, for application´s traffic the BIG-IP forwards the SNI received in the VIP.

Alternative monitoring in a 2-tier deployment

In this article, it is preferred to have 1:1 mapping between the L7 routes defined in the BIG-IP and the L7 routes defined in the ingress controller. It is possible to reduce the number of L7 routes and Service manifests for CIS by adding several monitors to a single L7 route manifest.

For example we could have a single base L7 route such as www.twotier.com/ and several monitors probing the different applications behind this FQDN, for example: www.twotier.com/, www.twotier.com/account, and www.twotier.com/shop.

This alternative monitoring can be configured with the following parameters:

In this example, you can see how to create such a configuration. 

The preference for using a 1:1 mapping instead is because although this alternative strategy creates less manifests, automating the creation of these might be more complex. Additionally, it will be less easy to know in a given moment which applications are not working fine.

Using OpenShift´s Router (HA-Proxy) in the second tier

CIS can be configured to ingest F5 CRs, OpenShift Routes CRs or Ingress resources. When CIS is configured for using Routes or Ingress, by default HA-Proxy will also ingest these. In other words, both CIS and HA-Proxy will implement the same L7 routes. This behavior also applies to single-tier deployments where the behavior could be desired for testing. In general this behavior is not desired.

HA-Proxy can be configured to do-not evaluate Routes or Ingress manifests with a given label or to do-not search for these in namespaces with a given label, for example "router=bigip". This behavior is configured in the IngressController resource type that manages HA-proxy's configuration:

{
  "spec": {
      "namespaceSelector": {
          "matchExpressions": [
              {
                  "key": "router",
                  "operator": "NotIn",
                  "values": [
                     "bigip"
                  ]
              }
          ]
      }
    }
}

This JSON configuration is applied with the next command for the default ingress controller/router in OpenShift:

oc -n openshift-ingress-operator patch ingresscontroller default --patch-file=router-default.shard.json --type=merge

More information on Route sharding can be found in this OpenShift document.

Using the default Router instance

OpenShift´s installer deploys the default HA-proxy instance with endpointPublishingStrategy type set to either hostNetwork or LoadBalancer. This will depend whether the OpenShift deployment is on-prem, in a public cloud, etc... This can be checked with the following command:

oc get ingresscontrollers default -n openshift-ingress-operator -o jsonpath='{.status.endpointPublishingStrategy}{"\n"}'

Regardless HA-proxy is configured with endpointPublishingStrategy either hostNetwork or LoadBalancer, CIS can be configured in clusterIP mode and refer to the HA-Proxy PODs using the "--orchestration-cni=ovn-k8s" and "--pool-member-type=cluster" options. The Service to refer to these HA-Proxy PODs will look as follows:

apiVersion: v1
kind: Service
metadata:
name: router-default-route-a
namespace: openshift-ingress
spec:
ports:
- name: http
  port: 80
  protocol: TCP
  targetPort: http
- name: https
  port: 443
  protocol: TCP
  targetPort: https
selector:
  ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default
  type: ClusterIP

Where as indicated earlier there will be a Service manifest for each L7 route in order to have per L7 route monitoring. Note that these Services reside in the openshift-ingress namespace, where the HA-Proxy PODs are.

Examples of this configuration can be found here using F5 CRs.

Using additional Router instances

Creating a new HA-Proxy instance just requires creating a new IngressController manifest in the openshift-ingress-operator namespace, such as the following sample:

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: shard-apps
namespace: openshift-ingress-operator
spec:
clientTLS:
defaultCertificate:
  name: router-shard-secret
domain: shard-apps.com
endpointPublishingStrategy:
type: Private
namespaceSelector:
  matchExpressions:
  - key: router
    operator: In
    values:
    - shard-apps
  replicas: 1

From this sample, note that:

• This new Router instance will evaluate only the L7 routes with namespaces with the label "router=shard-apps".
• the endpointPublishingStrategy is set to type Private, this is because we don´t need further exposure of it when CIS is accessing it in ClusterIP mode.

When using an additional HA-proxy instance, the default Router will need to be instructed not to evaluate these same L7 routes as the additional HA-proxy instance. This is shown next: 

{
  "spec": {
      "namespaceSelector": {
          "matchExpressions": [
              {
                  "key": "router",
                  "operator": "NotIn",
                  "values": [
                     "bigip",
                     "shard-apps"
                  ]
              }
          ]
      }
    }
}

An example of this configuration can be found here using Routes and multiple shards.

Using Istio Ingress Gateway / OpenShift Service Mesh

Using Istio (aka Service Mesh as packaged in OpenShift) in the second tier is straight-forward because Istio and CIS use different resource types for configuring L7 routes. A sample Service manifest to point to Istio ingress gateway is shown next

apiVersion: v1
kind: Service
metadata:
  name: svc-route-a
namespace: istio-system
spec:
ports:
- name: https
  port: 8443
  protocol: TCP
  targetPort: https
selector:
  app: istio-ingressgateway
type: ClusterIP

Where it can be observed that the Service manifests need to be created in the istio-system namespace and the selector needs to match the Istio ingress gateway´s Deployment's label. 

An example of this configuration can be found here using F5 CRs.

Using NGINX

Using NGINX in the second tier is much like using Istio, next is shown a sample Service manifest pointing to NGINX instances:

apiVersion: v1
kind: Service
metadata:
name: svc-route-a
namespace: nginx-ingress
spec:
ports:
- name: https
  port: 443
  protocol: TCP
  targetPort: 443
selector:
  app.kubernetes.io/instance: nginxingress-sample
  app.kubernetes.io/name: nginx-ingress
  type: ClusterIP

The only special consideration to be considered, and only if both NGINX and CIS are meant to use the Ingress resource type, is to specify in the Ingress manifests which controller will ingest the manifest. This is specified with the ingressClassName attribute as shown next:

in the case of NGINX

apiVersion: networking.k8s.io/v1 
kind: Ingress
[...]
spec:
 ingressClassName: nginx

or in the case of CIS

spec:
 ingressClassName: f5

An example of this configuration using F5 CRs in CIS and Ingress in NGINX can be found

Using advanced BIG-IP services with CIS

BIG-IP has many advanced services that can be easily referenced using F5 VirtualServer CRs, either directly in the manifest or referencing a Policy CR. When using Next Gen Routes these can also reference Policy CR but in a per router-group basis, hence having somewhat less granularity. Next is an overview of some of these advanced functionalities available.

The configuration of these services must be pre-created in the BIG-IP as profiles in the /Common partition which can be referenced freely by CIS. It is expected that CIS 2.17 will incorporate APM and SSLo.

Conclusion and closing remarks

F5 BIG-IP is an External Loadbalancer with application awareness which allows to do advanced L7 services (not just TLS termination) and unify the different ingress paths into the cluster, allowing to merge several ingress controllers, API managers, service mesh or a combination of these. This solution is ingress type agnostic.

A two-tier arrangement allows clean DevOps and NetOps separation. This is done by having separate L7 routes defined for the in-cluster ingress controllers and for the BIG-IPs.

To continue your journey, please check the examples in this GitHub repository using VirtualServer F5 CRs and OpenShift Route CRs.