Adopting SRE practices with F5: Multi-cluster Blue-green deployment

In last article, we covered blue-green deployment as the most straightforward SRE deployment model at a high level, here we are diving deeper into the details to see how F5 technologies enable this use case.

Let’s start off by looking at some of the key components.

F5 DNS Load Balancer Cloud Service (GSLB)

The first component of the solution is F5 Cloud Service. The DNS Load Balancer provides GSLB as a cloud-hosted SaaS service with built-in DDoS protection and an API-first approach. A blue-green deployment aims to minimize downtime due to app deployment, and there are some basic routing mechanisms out of the box with OpenShift that assist in this area. However, if we are looking for swift routing switch with more flexibility and reliability across different OpenShift clusters, different clouds, or geo locations, this is when F5 DNS Load Balancer Cloud Service comes into the picture.

Setting up DNS for F5 Cloud Services

This solution requires that your corporate DNS server delegates a DNS zone (aka subdomain) to the F5 DNS Load Balancer Cloud Service.

An OpenShift cluster typically has its own domain created for the applications, for example: *.apps.<cluster name>.example.com. The end user, however, doesn't really use such a long name and instead queries for www.example.com. A CNAME record is often used to map one domain name (an alias) to another (the true domain name).

All set up, this is the DNS scenario:

In case the customer has more than one cluster, it requires one CNAME record per cluster, with requests load balanced among clusters. The drawbacks of this type of solutions include:

No comprehensive health checking and monitoring
Unable to switch workloads across clusters at speed
Lack of automation and integration with the OpenShift cluster

F5 Cloud Services provides these features in a multi-cluster and multi-cloud infrastructure around the globe with the ease of a SaaS solution, without the need of infrastructure modifications. You will set up your corporate DNS to use F5 DNS Load Balancer Cloud Service as follows:

Here is a sample configuration for a Cloud/Corporate DNS:

You can register an F5 Cloud Service account, and then subscribe to DNS Load Balancer Sservice here:

F5 CloudServices

F5 GSLB tool for Ansible Automation

The blue-green deployment represents a sequence of steps to rollout your new application. GSLB tool is developed to provide a common automation plane for both OpenShift and F5 Cloud Service.

Leveraging the declarative API from F5 DNS Load Balancer Cloud Service and OpenShift, we used Ansible to automate the process. It enables you to standardize and automate release and deployment by orchestrating the build, test, provisioning, configuration management, and deployment tools in your Continuous Delivery pipeline.

More specifically, GSLB tool automates your interaction with:

the OpenShift/K8s deployments
Retrieve Layer 7 routes from given project/namespace and OpenShift Cluster
Copy Layer 7 routes of a given project/namespace from one OpenShift Cluster to another
F5 DNS Load Balancer Service
Create of GSLB load Balanced Records (LBRs) along with needed pieces (Monitors, IP endpoints, Pools etc.)
Set the GSLB ratio for each deployment for a given project/namespace

The benefits of using GSLB tool to automate the entire process:

Improve speed and scale especially with 100’s of OpenShift routes
Eliminate room for human error
Achieve deterministic and repeatable outcomes

I want to give credit to my colleague, Ulises Alonso Camaro, who developed the GSLB tool. Please refer to the GitHub for details of the GSLB tool, and wiki on how to set up the tool and operation.

Build and Run the Blue-green Deployment

Now we can look at how we can use F5 DNS Load Balancer Service and GSLB tool to canary test the new version and manipulate the traffic routing for Blue-green deployment. In Blue-green deployment, we deploy two versions of the application running simultaneously in two identical production environments called Blue (OpenShift Cluster 1) and Green (new OpenShift Cluster 2).

Step 1. Retrieve routes from Blue cluster and push to F5 DNS Load Balancer Cloud Service

Once you have installed the GSLB tool and configured the deployment settings for your infrastructure, the first set of commands to run are

./project-retrieve default aws1 && ./gslb-commit "publish routes from Blue cluster to F5 DNS load balancer"

These commands retrieve the OpenShift route(s) from your Blue cluster aws1, and then publish the retrieved routes into F5 DNS Load Balancer Cloud Service.

Step 2. Retrieve routes from Green cluster and push to F5 DNS Load Balancer Cloud Service

User to input the following commands:

./project-retrieve default aws2 &&./gslb-commit "publish routes from Blue cluster to F5 DNS load balancer"

These commands will retrieve the OpenShift route(s) from your Green cluster aws2, and push to F5 DNS Load Balancer Cloud Services configuration

Step 3. Canary test green deployment

User to input the following command:

./project-ratios default '{"aws1": "90", "aws2": "10" } && ./gslb-commit "canary testing blue and green clusters"

The commands will set the traffic ratio for the Blue (90%) and the Green deployment (10%) and publish the configuration.

As you can see, F5 DNS Load Balancer Cloud Service sets the traffic ratio for each endpoint accordingly.

Step 4. Switch traffic to Green

After the testing succeeds, it is time to switch production traffic to the Green cluster.

User to input the following commands:

./project-evacuate default aws1 && ./gslb-commit "switch all traffic to green cluster"

The commands will switch the traffic completely from the Blue to the Green deployment.

More Architectural Patterns

There are many related patterns for Blue-green deployment, each of which offers a different focus for an automated production deployment. Some example variants include:

Infrastructure as Code (IaC) In this variant of the pattern the release deployment target environment does not exist until it is created by the DevOps pipeline. Post deployment the original ‘blue’ environment is scheduled for destruction once the ‘green’ environment is considered stable in production.
Container-based Deployment In this variant of the pattern the release deployment target is represented as a collection of one or more containers. Post release, once the ‘green’ environment is considered stable in production, the containers represented by the ‘blue’ container group are scheduled for destruction.

Our solution can address all Blue-green deployment variants, with resources used in the blue and green environments can be created or destroyed as needed, or they can be geographically distributed.

While Continuous Deployment (CD) is a natural fit for the Blue-green deployment, F5 DNS Load Balancer Cloud Service combined with GSLB tool can enable many possibilities and support a collection of architecture patterns including:

Migrate application from a source cluster (OCP 3.x) to a destination cluster (OCP 4.x), refer here for details
Migrate workload from Kubernetes cluster to OpenShift Cluster
Modernize your application deployment with Lift and Shift. Repackage your application running as a set of VM’s into containers, and deploy then into OpenShift or Kubernetes cluster
Built into CI/CD pipeline so that any future changes to the application are built and deployed automatically.

We are continuously working on more usage patterns and will explore in more details in future blog posts.