Multi-Cluster, Multi-Cloud Networking for K8S with F5 Distributed Cloud – Architecture Pattern

Introduction

This is a two-part architecture pattern series

Part 1 – Architecture Pattern

Describe the problem statement, intended outcome, use cases and the Architecture Pattern of the solution.
Part 1 addresses “The Why” and “The What”.

Part 2 – Deployment Guide

Describe the deployment step of the solution.
Part 2 addresses “The How” and steps require to achieve the outcome.

Multi-Cluster, Multi-Cloud Networking (MCN) for Kubernetes with F5 Distributed Cloud (~15min)

Applications are part of our daily live. Indeed, it is part of our daily necessity. We use apps for digital wallet, check weather, navigation, online shopping and so on. We interact with it every mins or perhaps seconds. Application itself also transforming and evolving – from simple, static apps to complex, highly interactive over time. With the proliferation of various application framework, cloud and edge computing, distributed apps, microservices and so on, IT and application complexity has built up over time. This complexity has become an operational burden and essential threat to business success. According to F5’s State of Application Strategy 2022 (https://www.f5.com/state-of-application-strategy-report), the application landscape keeps changing and most organization manage 200-1,000 apps and 26% of app security and delivery technologies are distributed and deployed at the edge for efficiency outcomes.

Organization applications are constantly being transformed from traditional monolithic to modern microservices applications. Kubernetes platform such as F5 Distributed Cloud (F5 XC) AppStack, OpenShift Container Platform (OCP), EKS, AKS, GKE and etc is one of the key enabler for modern microservices application, which itself an important component of organization digital transformation strategy.

In this two-part architecture pattern document, I will share on how F5 XC with Secure Application Mesh Fabric essentially helps to simplify connectivity and security for your modern microservices applications.

Overview

“Any architect will know that a solid foundation is crucial to any building. Once the foundations are laid, necessary supports are put in place and permitting the rest of the structure to be built”. Success depends on a solid foundation. It is true as well for multi-cluster, multi-cloud/site networking for Kubernetes (K8S). Solid architecture pattern – “The Foundations”, are paramount for the success. An architecture pattern that fit for purpose, simple to consume, repeatable, secure and easy to operate that meet business and consumer requirements are important. Hence, the following architecture patterns for multi-cluster, multi-cloud/site networking for K8S will be described and explained.

Please do notes that this document is not about multi-cluster Kubernetes architecture design which commonly refer to single K8S cluster with single K8S control plane with distributed multiple worker nodes. This document related to multi-cluster K8S networking use cases (cluster to cluster) where each individual apps/pods distributed across multiple K8S cluster with each individual K8S control plane linked up or connected to form an application mesh network.

It is also important to acknowledge that generally it is a good practice to avoid “over-engineered” the entire solution that potentially may increases the complexities and Total Cost of Ownership (TCO). Hence, this pattern document may also be use as the baseline foundation for future additional use cases.

Why organization have multiple independent K8S cluster that distributed across many locations or networks?

Here are just a few common reasons

Regulatory requirement such as GDPR, PCI DSS, HIPPA or etc that require certain components of the apps to be hosted in country or on a specific controlled environment – DMZ zone, cloud, trusted and highly trusted zone.
Component of an apps may be managed and operated by different department within the same organization. For example, as part of merger and acquisition or line of business. Hence, it required cluster to be managed independently, yet need to be integrated.
Segregation of concern and application segmentation. Certain function within the overall application ecosystem needs to be managed by different team.
Application performance and optimization where apps (web frontend) required to be located closer to user (edge) where databases located in data center(s).
Distributed Apps + Cloud Migration Journey - Enterprises that located in different countries or multiple data center or clouds and could not runs apps on a single central K8S cluster.
Risk management – distribute application availability risk to different cluster rather than centralize in a single centralize cluster. It makes management of each individual cluster easier to manage compared to managing a centralize massive K8S cluster with a single control plane.

What are those common challenges arising with distributed individual K8S cluster?

Security – How to ensure confidentially, integrity and availability (CIA) between applications? Traditional network firewalls may not be able to address this modern distributed K8S clusters.
Visibility – Apps/Pods within a K8S egress the cluster and interact with apps/pods on another cluster. How to ensure adequate visibility, response time, error rate and etc. Organization commonly leverage API gateway to “glue” application together. With this API gateway strategy, what level of API visibility and security being implement? Are existing security controls sufficient to address the ever-increasing threat surface?
Complex networking setup between cluster.
Resiliency of application when cluster goes down. How to ensure seamless failover of application when cluster goes down?
High total cost of ownership (TCO) due to many clusters to manage and complex application interaction.
Lack of Skillset - complexities that drive the war for talent. Organization lack skills talent to manage, operate and secure.

How F5 Distributed Cloud Mesh able to helps to simplify connectivity and enhance security between multi-cluster apps?

Architecture Patterns

Architecture Pattern is a deployment scaffolding to address specific use cases. It purposes it to describe a recommendation and commonly accepted industry best practices way of solving a problem. Pattern represent a repeatable and reusable solution in decision making. It captures the design structure and the reasons behind and the desirable outcome. It intended purpose it to helps architect, solution designer, IT manager, subject matter expert and decision maker to avoid “over engineer” a particular solution, inconsistent implementation and to reduce operational complexities.

Note: OCP being use in this pattern document as the K8S platform. This pattern also applicable for Amazon EKS, Microsoft AKS and Google GKE.

Architecture Pattern 1

By default, when each VER pod(s) spin up (Volterra Edge Router), (single pod or cluster of three pods) it will auto register to F5 XC Regional Edge (RE) for control and management plane. These encrypted tunnels (dual for redundancy) can be used by VER pod(s) to mesh applications across K8S cluster. All K8S cluster network can have overlapping CIDR (Classless Inter-Domain Routing).

Cloud Mesh (also known as Customer Edge - CE) deployed as a Kubernetes Pods inside K8S cluster (e.g., OpenShift). This is also known as “Kubernetes sites” or “CE site on K8S”.
VER pod will perform automatic service discovery of pods within cluster. These capabilities enable VER pod to discover service endpoints and make it available to be consumed – within or across cluster. Automatic service discovery is important since service can be moved, scaled out, or published independently and in an automated way without coordinating with the proxy configuration.
Encrypted tunnel (IPSEC/SSL) can be established between VER pod to form a site mesh group – F5 XC Secure Application Mesh Fabric.
Pod(s) in Cluster1 can communicate with Pod(s) in Cluster 2 via this encrypted tunnel.
In the even if site mesh group tunnel not available, encrypted tunnel from control plane will be used as a backup data path to ensure availability of application.
Traffic delivery are native (leveraging existing K8S CNI) where VER pod send traffic to Kubernetes services via internal networking.
Suitable for DevSecOps where management and lifecycle of VER Pods(s) within the DevSec teams.

Pros

Cons

Cloud Mesh runs as a Kubernetes pods. Hence, no additional VM or bare metal server required.
Do not require additional service account for service discovery as VER pod live inside K8s and this enable VER pod to perform native service discovery
Easy to manage and operate.

This architecture pattern negates VER Pod(s) as a default egress gateway for K8S nodes. However, application can target domains or FQDNs that managed by VER Pod(s) and use that as an egress to others application resides on another cluster.
Establishment of encrypted tunnel (IPSEC/SSL) between VER Pod(s) uses K8S nodeport for L3 connectivity.
VER Pod(s) is a Kubernetes pod and it adhere to Kubernetes way of working. Traffic will not be able to reach VER Pod directly. It needs to be managed by an external load balancer to deliver traffic to VER Pod via a secure encrypted tunnel. F5 XC Regional Edge or a separate CE node can be used as a mechanism to channel traffic (Internet or Internal) to VER pod. Alternatively, access to VER Pods (s) via Kubernetes service nodeport.

Architecture Pattern 2

Cloud Mesh (CE node) deployed as a node/workload (VM or bare metal) outside K8S Cluster
Cloud Mesh node act as a secure ingress to K8S cluster or act as a Kubernetes egress gateway.
Cloud Mesh node require credential (e.g., service account) to query K8S API for service discovery.
Encrypted tunnel can be established among all nodes to form a site mesh group – F5 XC Secure Application Mesh Fabric.
Similarly with Pattern 1, Cloud Mesh control plane will be use as a redundant path in the even if the primary tunnel not available.
Traffic deliver are not native to K8S. F5 Mesh node deliver traffic to K8S pod via the following method and dependent on Kubernetes CNI.
- K8S Services runs as nodeport – Supported for all K8S variant
- Directly to K8S services/pods via OpenShift OVN-Kubernetes Advance CNI (static route from Cloud Mesh to OCP node-subnet via OCP node IP as gateway)
- Directly to K8S services/pods (e.g., EKS, AKS and GKE where application pods integrated to Cloud Provider networking)
- Directly to K8S services/pods via Calico BGP mesh
- Proxy to Kubernetes Ingress (e.g., OCP router, Nginx Ingress, etc)
Potentially suitable for NetSecOps or NetOps where management and lifecycle of Cloud Mesh Node within the NetOps teams.

Pros

Cons

Cloud Mesh node can be deployed as a Secure Kubernetes Gateway (SKG) – ingress and egress.
Ingress to the application can be delivered from any of the Cloud Mesh node. Doesn’t require additional node.
Cloud Mesh node(s) can be shared by multiple independent K8S cluster.
For OpenShift with OVN-Kubernetes CNI, ability to annotate namespace to use Cloud Mesh as an egress gateway (k8s.ovn.org/routing-external-gws)

Additional VM/bare metal server required. May need to manage and lifecycle (minimum 3 nodes for high availability) servers.
Require K8S service account to be loaded on Cloud Mesh to perform service discovery.
Application delivery to K8S pod or services (especially OpenShift) are limited and dependent on K8S CNI.

Architecture Pattern 3

Combination of Cloud Mesh as pod and Cloud Mesh as node
Inherit both pros and cons of Pattern 1 and Pattern 2.

Summary

Application is the center of attention for majority organization. It is an important asset for business. Application evolves from simple application to complex application with multitude of integrated systems and distributed – simple to complex. Application becoming so complex. Complexities are the real threat to business regardless from security or operation aspect. Security, clouds, multi-cloud networking and so on exist because of application. Those technologies will cease to exist without existence of application. F5 Distributed Cloud is design to address business most important asset - application. It bring F5 vision to reality for our customer - “Secure, Deliver and Optimize every app and API anywhere”.

In Part 2 – Deployment Guide, I am going to demonstrate and provides guidance on how you build the following demo environment base on sample Arcadia financial application and Nginx web server.