F5 BIG-IP deployment with Red Hat OpenShift - the no CIS option

 

Overview

There are deployments where F5 BIG-IP is used in front of a Red Hat OpenShift cluster without the Container Ingress Services (CIS) controller. In general, this is not recommended because without CIS there is no Kubernetes integration and automation. 

This article is for deployments where for some reason CIS is not used. The article complements the configuration outlined in OpenShift´s documentation section Load balancing requirements for user-provisioned infrastructure regardless deployed in on-prem, public-cloud, bare-metal or virtual machines.

When configuring a load balancer in front of OpenShift,  two types of scenarios need to be covered:

• (core) API load balancing: this refers to the Kubernetes API and the bootstrap (Machine Config Server) services. These are ports 6443 and 22623 respectively in the control-plane nodes.
  
  
• The application´s ingress. These are ports 443 and 80 pointing to the nodes where HA-Proxy instances (aka OpenShift´s router) are deployed. 
  
  Although not as critical as the core APIs, cluster critical applications are hosted through ingres. This is specially true for the HTTP route oauth-openshift.apps.<cluster name> required for authentication, but also in general for all OpenShift-deployed HTTP routes with name *.apps.<cluster name> .

Through this article it is assumed a three arm configuration, which is the more broad example. This is shown in the next figure where the BIG-IP has an Internal VLAN facing the OpenShift deployment, an Internet VLAN and a DMZ VLAN to face clients outside the Security domain of the OpenShift deployment.  

 

BIG-IP configuration for API endpoints - Overview

The configurations for both the Kubernetes API and the bootstrap services are very similar with the exception of the monitoring:

• The monitor for the Kubernetes API is an HTTPS monitor which checks both the HTTP 200 status code of /healthz and the "ok" in the payload for a successful probe. The interval, time-until-up and timeout follow Red Hat´s recommendations "Probing every 5 or 10 seconds, with two successful requests to become healthy and three to become unhealthy, are well-tested values".
  
  
• On the other hand, the monitor for the bootstrap (Machine Config Server) just checks for the HTTP status code of the same /healthz path checked (yet different port). All other settings remain the same.

The rest of the configuration is analogous for both (with the exception of the port used):

• The pool includes all the control-plane nodes, the just configured monitor and a reselect-retries value of 2, which helps in case a connection is sent to a down server before is marked unavailable. The number follows the recommendation of calculating this value as #pool members -1, for of small pools like this case.
  
  
• The virtual server is configured using a fastL4 type because no TLS offloading or advanced capabilities are going to be configured.
  
  The most relevant configuration in this case is the use of SNAT automap and restricting access to the virtual server, covered in the hardening section.
  
  

BIG-IP configuration for API endpoints - Configuration

In the next figure it is shown a sample virtual server, pool and health monitor configuration for the API virtual server. The configuration for the bootstrap virtual server is analogous (except the VS port and health check). The virtual server configuration is shown next:

Note that the above also includes source address configuration. This hardening is covered in a section further below. 

The pool and health monitor configuration follows:

Note that in the case of the monitor for the bootstrap service, the health monitor receive string doesn´t check for the "ok" string in the payload. Therefore the receive string is "^HTTP/1.1 200" instead.

BIG-IP configuration for Application Ingress - Overview

In OpenShift´s documentation Load balancing requirements for user-provisioned infrastructure it is advised that for application´s ingress just a L4 load balancer with source address persistence should be used. In this article, It is considered that configuration is just the least common denominator across load balancers. Configuring the virtual server as standard full-proxy, can give benefits such as:

• SSL/TLS off-loading with SSL/TLS accelerators or by terminating the SSL/TLS in the BIG-IP
• Allows the use of SSL/TLS Hardware Secure Modules (HSM) or Network HSMs to securely store SSL/TLS keys.
• Allows to handle application layer data, allowing for content inspection, content-based switching, content-insertion or rewrite, either in the headers or in the payload.
• Allows to use BIG-IP's application layer security features such as Advanced WAF, Access Policy Manager or Bot defense.
• Inserting the simple but crucial task of inserting the X-Forwarded-For header.
• Allows for HTTP Cookie session persistence when the source IP (or hash) of the client cannot be used for persistence. 
• Allows to restrict access to certain applications based on FQDN and path. For example, *.apps.<cluster name>.

If any of the of the above features are desired, it is recommended to create two sets of virtual servers:

• A set of HTTP/HTTPS L4 virtual servers for the routes with name *.apps.<cluster name>. This way, applications see the expected clusters certificates. 
  
  
• For custom routes, for example myapp.mydomain.com create separated HTTP/HTTPS virtual servers using the standard virtual server type. Optionally these routes could be in a separate shard as indicated in the hardening section. 

The next is a set of recommendations for appropriate health checking and failure handling of HA-proxy instances in OpenShift: 

• Create an HTTP monitor for the readiness endpoint
  
  When a POD is not in Ready state, CIS automatically removes it from the pool. When not using CIS we need to mimic this functionality with an HTTP monitor in port 1936 (readiness endpoint) for both 443 and 80-port pools:
  
  

  ltm monitor http ocp_haproxy 
  {   
     destination *.1936
     recv "^HTTP/1.1 200"
     send "GET /healthz/ready HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n" 
  }
  

  
  This monitor will detect the following failure scenarios:
  
  
  • There is no IP connectivity to the HA-proxy instance
  • HA-proxy is not working properly (it will return a non-HTTP 200 status code)
  • HA-proxy is performing a graceful shutdown (a POD deletion), for example when doing an upgrade (returning an HTTP 500 status code).
  
  The graceful shutdown process of HA-proxy is as follows:
  
  
  • the readiness endpoint immediately returns an HTTP 500 error but will continue processing requests for the applications for 45 seconds. This is indicated in the response of the readiness endpoint with an ”[+]backend-http ok” message in the payload.
    
    
  • after 45 seconds, the endpoint will return “[-]backend-http failed: reason withheld” and will TCP RESET any request for the applications. During these 45 seconds, the timer values to the HTTP monitor have plenty of time to disable the HA-proxy instance appropriately.

The above health monitor is applied for both HTTP and HTTPS pools. By default, any worker node can be selected for the placement of the ingress controllers PODs. This can be controlled with the nodePlacement attribute of the IngressController resource. In the pool´s configuration (one for each port), the following configuration is advised:

• Handle non-graceful errors with the pool's reselect tries 
  
  To deal better with non-graceful shutdowns or transient errors, we can make use of this mechanism which will reselect a new HA-proxy instance when a request to an application fails. The recommendation is to set the number of tries to the number of HA-proxy instances -1.

• Set an additional TCP monitor for HA-proxy´s application's path socket
  
  This additional TCP monitor in either port 80 or 443 complements the HTTP monitor of the readiness endpoint by additionally validating that the HA-proxy instances can listen for requests in their designated application's socket. Although this is handled with the reselect tries mechanism this monitor will provide visibility that such types of errors are happening.

• Established connections
  
  For already established connections the Action on Service Down feature could be used but it is considered that the default option NONE should be used. This will help finishing already established connections.  Using RESET would trigger a faster retry from the HTTP client with the disadvantage of not allowing finishing requests that could potentially finish. Other options are not worth considering.

Regardless of the virtual server type chosen (fastL4 or standard), when the BIG-IP is used in one-arm mode will require the use SNAT. On the other hand, when using two-arm mode and the traffic is reached through BIG-IP's external network, the use of SNAT can be avoided by using AdminPolicyBasedExternalRoute (see below the section Keeping the source IP of ingress clients below.

 

BIG-IP configuration for Ingress - Configuration

In the next image, it can be seen graphically the configuration recommended with key settings highlighted for the pool and monitor.

In this section it is not shown a specific configuration for the virtual servers because these depend on the routing of the deployment, features and hardening desired (see next sections for details), yet the following must be always setup:

• When using L4 virtual servers, apply the default source address persistence profile.
  
  
• When using standard type virtual servers, apply the default cookie persistence profile. 

Virtual Server configuration hardening

Simple but yet effective way of hardening these virtual servers can be done in several ways:

• Specifying explicitly in which VLANs the virtual servers listen for requests. In the case of the bootstrap service this should be only the VLAN between the BIG-IP and the OpenShift cluster. In the case of the Kubernetes API, this should include the same VLAN but potentially there might be Kubernetes clients accessing through a different BIG-IP VLAN.
  
  
• Specifying the source addresses expected to reach the virtual servers. In the case of the bootstrap service this should be set to the subnet of the OpenShift nodes. In the case of the Kubernes API this will be highly dependent 

Some organisations might choose not to specify source addresses and instead rely in firewalls to control the flows reaching the virtual services. Specifying the VLANs should be considered a minimum best practice.

Additionally, this article suggests that application ingress should be segregated in ingress route shards (separate OpenShift router instances) dedicated for each purpose, for example: *.apps.<cluster name> , internal applications and external apps. Alternatively, access restriction can be done as well in a single virtual server in the BIG-IP by using LTM policy rules.

SNATs and keeping the source IP of ingress clients

By default OpenShift sends the traffic to the default gateway of the network where the OpenShift cluster is placed. Traditionally, in the case of ingress traffic to the applications, in order to make the traffic symmetric (making the return traffic go through the load balancer) it has been required to use SNAT in the load balancer.

With the introduction AdminPolicyBasedExternalRoute this is no longer required, allowing the use of the source IP to the workload PODs (in the case of 1-tier deployment) or to the in-cluster ingress controller (in the case of 2-tier deployments). The overall traffic flows in the platform would be as shown next:

 

Using the feature is straight forward, it is noly required the namespaces for which we want the BIG-IP to be the gateway and the floating IP of the BIG-IPs facing the OpenShift cluster. An example is shown next: 

apiVersion: k8s.ovn.org/v1
kind: AdminPolicyBasedExternalRoute
metadata:
name: meg-policy-common
spec:
from:
  namespaceSelector:
    matchLabels:
      meg: common
nextHops:
  static:
    - ip: "10.1.20.100"

Where 10.1.20.100 represents the internal floating IP of the BIG-IP facing the OpenShift cluster. The labels can be freely chosen. More details in using this OpenShift feature can be found in this article.

Please note that all the above applies for external (dmz or Internet connections). In the case of connections from the internal VLAN it is required to use SNAT.

In any case, when using SNAT, it might be required to configure SNAT pools instead of using SNAT automap to avoid port exhaustion . The more likely virtual server to require SNAT pools is the one for the customer apps. It might be also convenient have separate SNAT pools depending of the origin of the client, or the listener VLAN used. These SNAT pools can be configured by means of using LTM policies.

A sample SNAT configuration can be shown in the next figure.

Conclusions

F5 BIG-IP can be configured for control plane and applications load balancing of your OpenShift clusters. It allows fine grained access control, scalable SNAT or keeping original source IP amongst others. It also provides enhanced L7 features which allows additional security, TLS off-loading, TLS HSM support and traffic manipulation.

Last but not least, F5 BIG-IP also has sophisticated features for health checking and handling failures to operate without any service disruption. 

We look forward to know your experience and feedback on this article.