Deploy OpenShift 4.x with BIG-IP CIS in AWS

OpenShift Container Platform (or OCP) provides the HAProxy template router as the default plug-in as the ingress point for all external traffic. While this is fine for small scale deployments there are some significant challenges when looking to scale your OCP deployments beyond single cluster, single site deployments. As with any architectural design, we have to consider our desired ‘end state’ architecture. For example: 

• Will your organization deploy applications across clusters as the environment starts to scale?
• How about agile development methodologies and blue/green A/B deployment scenarios, will the default ADC have the intelligence to automatically direct traffic between production and non-production workloads?
• How about failover and site resiliency?

F5 BIG-IP provides these services using Container Ingress Services or CIS, with a more simplified architecture, to help your organization scale applications and services across clusters and sites. In addition, F5 BIG-IP offers advanced access and security control for the traffic going into or out of an OpenShift cluster, to ensure consistent policy enforcement and end to end compliance in any cloud. In this article, we're going to walk you through a fairly minimum deployment of OpenShift 4.3 with BIG-IP CIS in Amazon Web Services (AWS). With such, you can enable more complex use cases.

So let’s get started.

Prerequisites

• If you do not have an AWS profile stored on your computer, enter the AWS access key ID and secret access key for the user that you configured to run the installation program.
• Confirm AWS IAM user name that you are using to create OpenShift cluster is granted the AdministratorAccess policy. 
• Make sure you have the access the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site. If you have a Red Hat account, log in with your credentials. If you do not, create an account.

Configuring Route53 

To install OpenShift Container Platform, the AWS account you use must have a dedicated public hosted zone in your Route53 service. This zone must be authoritative for the domain. The Route53 service provides cluster DNS resolution and name lookup for external connections to the OCP cluster.

If you registered domain with Route53, you do not need any further configuration as a hosted zone was automatically created. If you use public domain hosted outside Route53, you would need do the following:

1. Create a public hosted zone for your domain or subdomain. See Creating a Public Hosted Zone in the AWS documentation.
2. Shared the NS record and SA record with your IT team for adding the entries in DNS. 

Provision OpenShift cluster

Before you install OpenShift Container Platform, download the installation file on a local computer. 

1. Access the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site.
2. Navigate to the page for your installation type, download the installation program for your operating system, and place the file in the directory where you will store the installation configuration files.
3. The installation program creates several files on the computer that you use to install your cluster. You must keep both the installation program and the files that the installation program creates after you finish installing the cluster.
4. Extract the installation program. For example, on a computer that uses a Linux operating system, run the following command: tar xvf <installation_program>.tar.gz
5. From the Pull Secret page on the Red Hat OpenShift Cluster Manager site, download your installation pull secret as a .txt file. This pull secret allows you to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift Container Platform components. 

Run the installation program: 

❯ ./openshift-install create cluster --dir ~/aws-ocp43 --log-level=info
? SSH Public Key /Users/zji/.ssh/id_rsa.pub
? Platform aws
? Region us-west-2
? Base Domain <mybasedomain>
? Cluster Name cluster1
? Pull Secret [? for help] 
*********************************************************************************
INFO Creating infrastructure resources
INFO Waiting up to 30m0s for the Kubernetes API at https://api.cluster1.mybasedomain:6443...
INFO API v1.16.2+f2384e2 up
INFO Waiting up to 30m0s for bootstrapping to complete...
INFO Destroying the bootstrap resources...
INFO Waiting up to 30m0s for the cluster at https://api.cluster1.mybasedomain:6443 to initialize...
INFO Waiting up to 10m0s for the openshift-console route to be created...
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/Users/zji/aws-ocp43/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.cluster1.mybasedomain
INFO Login to the console with user: kubeadmin, password: 00000-00000-00000-00000 

What just happened?

Let's review what just happened. The above installation program automatically set up the following AWS resources for Red Hat OpenShift environment:

• A virtual private cloud (VPC) that spans three Availability Zones, with one private and one public subnet in each Availability Zone.
• An internet gateway to provide internet access to each subnet.
• An OpenShift master ELB
• An OpenShift node ELB
• In the private subnets:
• Three OpenShift master (including etcd) instances in an Auto Scaling group
• Three OpenShift node instances in an Auto Scaling group

Source: https://aws.amazon.com/quickstart/architecture/openshift/

As an account admin for AWS, you can list all these resources that OpenShift or its installer has created per cluster.

❯ aws resourcegroupstaggingapi get-resources --tag-filters "Key=kubernetes.io/cluster/cluster2-7j2jr" | jq '.ResourceTagMappingList[].ResourceARN'
"arn:aws:ec2:us-west-2:877162104333:dhcp-options/dopt-0d8651a54eddb2acb"
"arn:aws:ec2:us-west-2:877162104333:elastic-ip/eipalloc-0c4b4d66dbf695655"
"arn:aws:ec2:us-west-2:877162104333:elastic-ip/eipalloc-077f8efc0cd8d0b01"
"arn:aws:ec2:us-west-2:877162104333:elastic-ip/eipalloc-05001638bc043f0cd"
"arn:aws:ec2:us-west-2:877162104333:elastic-ip/eipalloc-03abd4c3fb87a7a7d"
...

Logging in to the cluster 

Next, you can install the CLI in order to interact with OpenShift Container Platform using a command-line interface. You can log in to your cluster as a default system user by exporting the cluster kubeconfig file 

• Export the kubeadmin credentials:

$ export KUBECONFIG=<installation_directory>/auth/kubeconfig

• Verify you can run oc commands successfully using the exported configuration:

$ oc whoami 
kube:admin
❯ oc get node 
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-0-128-147.us-west-2.compute.internal   Ready    worker   26m   v1.16.2+f2384e2
ip-10-0-141-160.us-west-2.compute.internal   Ready    master   34m   v1.16.2+f2384e2
ip-10-0-149-163.us-west-2.compute.internal   Ready    master   34m   v1.16.2+f2384e2 
ip-10-0-152-36.us-west-2.compute.internal    Ready    worker   26m   v1.16.2+f2384e2
ip-10-0-160-247.us-west-2.compute.internal   Ready    master   34m   v1.16.2+f2384e2
ip-10-0-169-120.us-west-2.compute.internal   Ready    worker   25m   v1.16.2+f2384e2

Simplify Load Balancing with BIG-IP

By default, OpenShift deployment instantiates the build-in HAProxy template router as the default router. For OpenShift in AWS, it also deploys an AWS ELB as the frontend L4 load balancer, resulting in a two-layer load balancer architecture as illustrated below. Some patterns insert yet another layer of scalability across clusters.

F5 BIG-IP simplifies the architecture with a single layer of load balancer where the BIG-IP is exposed directly to the Internet and also performs L7 routing including SSL off-loading, thus improves performance of apps served from the cluster and scalability of the overall architecture. 

It also offers additional benefits. You can further reduce latency by adding Advanced WAF, Access Policy control, intelligence traffic management, many more application delivery and security offerings by BIG-IP.

Follow the steps to deploy BIG-IP into existing VPC:

https://clouddocs.f5.com/cloud/public/v1/aws_index.html

Next, you can refer to F5 CIS user guide to deploy and configure CIS for OpenShift.

If you deploy BIG-IP CIS as cluster mode, you may implement VXLAN to route the traffic between BIG-IP and OpenShift Cluster. By default, direct access to OpenShift nodes is limited. To support VXLAN traffic from BIG-IP, you want to adjust the OpenShift security group accordingly by exposing additional ports as following: 

You can verify that F5 BIG-IP CIS is successfully installed:

❯ oc get pods -n kube-system -o wide
NAME               READY  STATUS  RESTARTS  AGE  IP      NODE                     NOMINATED NODE  READINESS GATES
k8s-bigip-ctlr-6664d45f57-cjb8g  1/1   Running  0     15d  10.131.0.46  ip-10-0-222-250.us-west-2.compute.internal  <none>      <none>

Summary

Red Hat provides an excellent foundation for building a production ready OpenShift in AWS environment, BIG-IP CIS can further simplify the architecture and improve performance by converging the 2-tier load balancing into single layer. In addition, BIG-IP can provide advanced application delivery and security features, and we will cover more use cases in the following articles.