on 11-Aug-2020 15:09
OpenShift Container Platform (or OCP) provides the HAProxy template router as the default plug-in as the ingress point for all external traffic. While this is fine for small scale deployments there are some significant challenges when looking to scale your OCP deployments beyond single cluster, single site deployments. As with any architectural design, we have to consider our desired ‘end state’ architecture. For example:
F5 BIG-IP provides these services using Container Ingress Services or CIS, with a more simplified architecture, to help your organization scale applications and services across clusters and sites. In addition, F5 BIG-IP offers advanced access and security control for the traffic going into or out of an OpenShift cluster, to ensure consistent policy enforcement and end to end compliance in any cloud. In this article, we're going to walk you through a fairly minimum deployment of OpenShift 4.3 with BIG-IP CIS in Amazon Web Services (AWS). With such, you can enable more complex use cases.
So let’s get started.
To install OpenShift Container Platform, the AWS account you use must have a dedicated public hosted zone in your Route53 service. This zone must be authoritative for the domain. The Route53 service provides cluster DNS resolution and name lookup for external connections to the OCP cluster.
If you registered domain with Route53, you do not need any further configuration as a hosted zone was automatically created. If you use public domain hosted outside Route53, you would need do the following:
Before you install OpenShift Container Platform, download the installation file on a local computer.
Run the installation program:
❯ ./openshift-install create cluster --dir ~/aws-ocp43 --log-level=info ? SSH Public Key /Users/zji/.ssh/id_rsa.pub ? Platform aws ? Region us-west-2 ? Base Domain <mybasedomain> ? Cluster Name cluster1 ? Pull Secret [? for help] ********************************************************************************* INFO Creating infrastructure resources INFO Waiting up to 30m0s for the Kubernetes API at https://api.cluster1.mybasedomain:6443... INFO API v1.16.2+f2384e2 up INFO Waiting up to 30m0s for bootstrapping to complete... INFO Destroying the bootstrap resources... INFO Waiting up to 30m0s for the cluster at https://api.cluster1.mybasedomain:6443 to initialize... INFO Waiting up to 10m0s for the openshift-console route to be created... INFO Install complete! INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/Users/zji/aws-ocp43/auth/kubeconfig' INFO Access the OpenShift web-console here: https://console-openshift-console.apps.cluster1.mybasedomain INFO Login to the console with user: kubeadmin, password: 00000-00000-00000-00000
Let's review what just happened. The above installation program automatically set up the following AWS resources for Red Hat OpenShift environment:
Source: https://aws.amazon.com/quickstart/architecture/openshift/
As an account admin for AWS, you can list all these resources that OpenShift or its installer has created per cluster.
❯ aws resourcegroupstaggingapi get-resources --tag-filters "Key=kubernetes.io/cluster/cluster2-7j2jr" | jq '.ResourceTagMappingList[].ResourceARN' "arn:aws:ec2:us-west-2:877162104333:dhcp-options/dopt-0d8651a54eddb2acb" "arn:aws:ec2:us-west-2:877162104333:elastic-ip/eipalloc-0c4b4d66dbf695655" "arn:aws:ec2:us-west-2:877162104333:elastic-ip/eipalloc-077f8efc0cd8d0b01" "arn:aws:ec2:us-west-2:877162104333:elastic-ip/eipalloc-05001638bc043f0cd" "arn:aws:ec2:us-west-2:877162104333:elastic-ip/eipalloc-03abd4c3fb87a7a7d" ...
Next, you can install the CLI in order to interact with OpenShift Container Platform using a command-line interface. You can log in to your cluster as a default system user by exporting the cluster kubeconfig file
$ export KUBECONFIG=<installation_directory>/auth/kubeconfig
$ oc whoami kube:admin ❯ oc get node NAME STATUS ROLES AGE VERSION ip-10-0-128-147.us-west-2.compute.internal Ready worker 26m v1.16.2+f2384e2 ip-10-0-141-160.us-west-2.compute.internal Ready master 34m v1.16.2+f2384e2 ip-10-0-149-163.us-west-2.compute.internal Ready master 34m v1.16.2+f2384e2 ip-10-0-152-36.us-west-2.compute.internal Ready worker 26m v1.16.2+f2384e2 ip-10-0-160-247.us-west-2.compute.internal Ready master 34m v1.16.2+f2384e2 ip-10-0-169-120.us-west-2.compute.internal Ready worker 25m v1.16.2+f2384e2
By default, OpenShift deployment instantiates the build-in HAProxy template router as the default router. For OpenShift in AWS, it also deploys an AWS ELB as the frontend L4 load balancer, resulting in a two-layer load balancer architecture as illustrated below. Some patterns insert yet another layer of scalability across clusters.
F5 BIG-IP simplifies the architecture with a single layer of load balancer where the BIG-IP is exposed directly to the Internet and also performs L7 routing including SSL off-loading, thus improves performance of apps served from the cluster and scalability of the overall architecture.
It also offers additional benefits. You can further reduce latency by adding Advanced WAF, Access Policy control, intelligence traffic management, many more application delivery and security offerings by BIG-IP.
Follow the steps to deploy BIG-IP into existing VPC:
https://clouddocs.f5.com/cloud/public/v1/aws_index.html
Next, you can refer to F5 CIS user guide to deploy and configure CIS for OpenShift.
If you deploy BIG-IP CIS as cluster mode, you may implement VXLAN to route the traffic between BIG-IP and OpenShift Cluster. By default, direct access to OpenShift nodes is limited. To support VXLAN traffic from BIG-IP, you want to adjust the OpenShift security group accordingly by exposing additional ports as following:
You can verify that F5 BIG-IP CIS is successfully installed:
❯ oc get pods -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES k8s-bigip-ctlr-6664d45f57-cjb8g 1/1 Running 0 15d 10.131.0.46 ip-10-0-222-250.us-west-2.compute.internal <none> <none>
Red Hat provides an excellent foundation for building a production ready OpenShift in AWS environment, BIG-IP CIS can further simplify the architecture and improve performance by converging the 2-tier load balancing into single layer. In addition, BIG-IP can provide advanced application delivery and security features, and we will cover more use cases in the following articles.
Great article Eric. I appreciate the detail in what's required to deploy OCP. I'm looking forward to further articles covering use cases.