Red Hat OpenShift

10 Topics

F5 BIG-IP deployment with Red Hat OpenShift - keeping client IP addresses and egress flows
Controlling the egress traffic in OpenShift allows to use the BIG-IP for several use cases: Keeping the source IP of the ingress clients Providing highly scalable SNAT for egress flows Providing security functionalities for egress flows
Ulises_Alonso
Mar 15, 2025 Place Technical Articles
431Views
1like
1Comment
Secure AI RAG using F5 Distributed Cloud in Red Hat OpenShift AI and NetApp ONTAP Environment
Introduction Retrieval Augmented Generation (RAG) is a powerful technique that allows Large Language Models (LLMs) to access information beyond their training data. The “R” in RAG refers to the data retrieval process, where the system retrieves relevant information from an external knowledge base based on the input query. Next, the “A” in RAG represents the augmentation of context enrichment, as the system combines the retrieved relevant information and the input query to create a more comprehensive prompt for the LLM. Lastly, the “G” in RAG stands for response generation, where the LLM generates a response with a more contextually accurate output based on the augmented prompt as a result. RAG is becoming increasingly popular in enterprise AI applications due to its ability to provide more accurate and contextually relevant responses to a wide range of queries. However, deploying RAG can introduce complexity due to its components being located in different environments. For instance, the datastore or corpus, which is a collection of data, is typically on-premise for enhanced control over data access and management due to data security, governance, and compliance with regulations within the enterprise. Meanwhile, inference services are often deployed in the cloud for their scalability and cost-effectiveness. In this article, we will discuss how F5 Distributed Cloud can simplify the complexity and securely connect all RAG components seamlessly for enterprise RAG-enabled AI applications deployments. Specifically, we will focus on Network Connect, App Connect, and Web App & API Protection. We will demonstrate how these F5 Distributed Cloud features can be leveraged to secure RAG in collaboration with Red Hat OpenShift AI and NetApp ONTAP. Example Topology F5 Distributed Cloud Network Connect F5 Distributed Cloud Network Connect enables seamless and secure network connectivity across hybrid and multicloud environments. By deploying F5 Distributed Cloud Customer Edge (CE) at site, it allows us to easily establish encrypted site-to-site connectivity across on-premises, multi-cloud, and edge environment. Jensen Huang, CEO of NVIDIA, has said that "Nearly half of the files in the world are stored on-prem on NetApp.”. In our example, enterprise data stores are deployed on NetApp ONTAP in a data center in Seattle managed by organization B (Segment-B: s-gorman-production-segment), while RAG services, including embedding Large Language Model (LLM) and vector database, is deployed on-premise on a Red Hat OpenShift cluster in a data center in California managed by Organization A (Segment-A: jy-ocp). By leveraging F5 Distributed Cloud Network Connect, we can quickly and easily establish a secure connection for seamless and efficient data transfer from the enterprise data stores to RAG services between these two segments only: F5 Distributed Cloud CE can be deployed as a virtual machine (VM) or as a pod on a Red Hat OpenShift cluster. In California, we deploy the CE as a VM using Red Hat OpenShift Virtualization — click here to find out more on Deploying F5 Distributed Cloud Customer Edge in Red Hat OpenShift Virtualization: Segment-A: jy-ocp on CE in California and Segment-B: s-gorman-production-segment on CE in Seattle: Simply and securely connect Segment-A: jy-ocp and Segment-B: s-gorman-production-segment only, using Segment Connector: NetApp ONTAP in Seattle has a LUN named “tbd-RAG”, which serves as the enterprise data store in our demo setup and contains a collection of data. After these two data centers are connected using F5 XC Network Connect, a secure encrypted end-to-end connection is established between them. In our example, “test-ai-tbd” is in the data center in California where it hosts the RAG services, including embedding Large Language Model (LLM) and vector database, and it can now successfully connect to the enterprise data stores on NetApp ONTAP in the data center in Seattle: F5 Distributed Cloud App Connect F5 Distributed Cloud App Connect securely connects and delivers distributed applications and services across hybrid and multicloud environments. By utilizing F5 Distributed Cloud App Connect, we can direct the inference traffic through F5 Distributed Cloud's security layers to safeguard our inference endpoints. Red Hat OpenShift on Amazon Web Services (ROSA) is a fully managed service that allows users to develop, run, and scale applications in a native AWS environment. We can host our inference service on ROSA so that we can leverage the scalability, cost-effectiveness, and numerous benefits of AWS’s managed infrastructure services. For instance, we can host our inference service on ROSA by deploying Ollama with multiple AI/ML models: Or, we can enable Model Serving on Red Hat OpenShift AI (RHOAI). Red Hat OpenShift AI (RHOAI) is a flexible and scalable AI/ML platform builds on the capabilities of Red Hat OpenShift that facilitates collaboration among data scientists, engineers, and app developers. This platform allows them to serve, build, train, deploy, test, and monitor AI/ML models and applications either on-premise or in the cloud, fostering efficient innovation within organizations. In our example, we use Red Hat OpenShift AI (RHOAI) Model Serving on ROSA for our inference service: Once inference service is deployed on ROSA, we can utilize F5 Distributed Cloud to secure our inference endpoint by steering the inference traffic through F5 Distributed Cloud's security layers, which offers an extensive suite of features designed specifically for the security of modern AI/ML inference endpoints. This setup would allow us to scrutinize requests, implement policies for detected threats, and protect sensitive datasets before they reach the inferencing service hosted within ROSA. In our example, we setup a F5 Distributed Cloud HTTP Load Balancer (rhoai-llm-serving.f5-demo.com), and we advertise it to the CE in the datacenter in California only: We now reach our Red Hat OpenShift AI (RHOAI) inference endpoint through F5 Distributed Cloud: F5 Distributed Cloud Web App & API Protection F5 Distributed Cloud Web App & API Protection provides comprehensive sets of security features, and uniform observability and policy enforcement to protect apps and APIs across hybrid and multicloud environments. We utilize F5 Distributed Cloud App Connect to steer the inference traffic through F5 Distributed Cloud to secure our inference endpoint. In our example, we protect our Red Hat OpenShift AI (RHOAI) inference endpoint by rate-limiting the access, so that we can ensure no single client would exhaust the inference service: A "Too Many Requests" is received in the response when a single client repeatedly requests access to the inference service at a rate higher than the configured threshold: This is just one of the many security features to protect our inference service. Click here to find out more on Securing Model Serving in Red Hat OpenShift AI (on ROSA) with F5 Distributed Cloud API Security. Demonstration In a real-world scenario, the front-end application could be hosted on the cloud, or hosted at the edge, or served through F5 Distributed Cloud, offering flexible alternatives for efficient application delivery based on user preferences and specific needs. To illustrate how all the discussed components work seamlessly together, we simplify our example by deploying Open WebUI as the front-end application on the Red Hat OpenShift cluster in the data center in California, which includes RAG services. While a DPU or GPU could be used for improved performance, our setup utilizes a CPU for inferencing tasks. We connect our app to our enterprise data stores deployed on NetApp ONTAP in the data center in Seattle using F5 Distributed Cloud Network Connect, where we have a copy of "Chapter 1. About the Migration Toolkit for Virtualization" from Red Hat. These documents are processed and saved to the Vector DB: Our embedding Large Language Model (LLM) is Sentence-Transformers/all-MiniLM-L6-v2, and here is our RAG template: Instead of connecting to the inference endpoint on Red Hat OpenShift AI (RHOAI) on ROSA directly, we connect to the F5 Distributed Cloud HTTP Load Balancer (rhoai-llm-serving.f5-demo.com) from F5 Distributed Cloud App Connect: Previously, we asked, "What is MTV?“ and we never received a response related to Red Hat Migration Toolkit for Virtualization: Now, let's try asking the same question again with RAG services enabled: We finally received the response we had anticipated. Next, we use F5 Distributed Cloud Web App & API Protection to safeguard our Red Hat OpenShift AI (RHOAI) inference endpoint on ROSA by rate-limiting the access, thus preventing a single client from exhausting the inference service: As expected, we received "Too Many Requests" in the response on our app upon requesting the inference service at a rate greater than the set threshold: With F5 Distributed Cloud's real-time observability and security analytics from the F5 Distributed Console, we can proactively monitor for potential threats. For example, if necessary, we can block a client from accessing the inference service by adding it to the Blocked Clients List: As expected, this specific client is now unable to access the inference service: Summary Deploying and securing RAG for enterprise RAG-enabled AI applications in a multi-vendor, hybrid, and multi-cloud environment can present complex challenges. In collaboration with Red Hat OpenShift AI (RHOAI) and NetApp ONTAP, F5 Distributed Cloud provides an effortless solution that secures RAG components seamlessly for enterprise RAG-enabled AI applications.
Jennifer_Yeung
Feb 25, 2025 Place Technical Articles
304Views
1like
0Comments
Introducing F5 BIG-IP Next CNF Solutions for Red Hat OpenShift
5G and Red Hat OpenShift 5G standards have embraced Cloud-Native Network Functions (CNFs) for implementing network services in software as containers. This is a big change from previous Virtual Network Functions (VNFs) or Physical Network Functions (PNFs). The main characteristics of Cloud-Native Functions are: Implementation as containerized microservices Small performance footprint, with the ability to scale horizontally Independence of guest operating system, since CNFs operate as containers Lifecycle manageable by Kubernetes Overall, these provide a huge improvement in terms of flexibility, faster service delivery, resiliency, and crucially using Kubernetes as unified orchestration layer. The later is a drastic change from previous standards where each vendor had its own orchestration. This unification around Kubernetes greatly simplifies network functions for operators, reducing cost of deploying and maintaining networks. Additionally, by embracing the container form factor, allows Network Functions (NFs) to be deployed in new use cases like far edge. This is thanks to the smaller footprint while at the same time these can be also deployed at large scale in a central data center because of the horizontal scalability. In this article we focus on Red Hat OpenShift which is the market leading and industry reference implementation of Kubernetes for IT and Telco workloads. Introduction to F5 BIG-IP Next CNF Solutions F5 BIG-IP Next CNF Solutions is a suite of Kubernetes native 5G Network Functions, implemented as microservices. It shares the same Cloud Native Engine (CNE) as F5 BIG-IP Next SPK introduced last year. The functionalities implemented by the CNF Solutions deal mainly with user plane data. User plane data has the particularity that the final destination of the traffic is not the Kubernetes cluster but rather an external end-point, typically the Internet. In other words, the traffic gets in the Kubernetes cluster and it is forwarded out of the cluster again. This is done using dedicated interfaces that are not used for the regular ingress and egress paths of the regular traffic of a Kubernetes cluster. In this case, the main purpose of using Kubernetes is to make use of its orchestration, flexibility, and scalability. The main functionalities implemented at initial GA release of the CNF Solutions are: F5 Next Edge Firewall CNF, an IPv4/IPv6 firewall with the main focus in protecting the 5G core networks from external threads, including DDoS flood protection and IPS DNS protocol inspection. F5 Next CGNAT CNF, which offers large scale NAT with the following features: NAPT, Port Block Allocation, Static NAT, Address Pooling Paired, and Endpoint Independent mapping modes. Inbound NAT and Hairpining. Egress path filtering and address exclusions. ALG support: FTP/FTPS, TFTP, RTSP and PPTP. F5 Next DNS CNF, which offers a transparent DNS resolver and caching services. Other remarkable features are: Zero rating DNS64 which allows IPv6-only clients connect to IPv4-only services via synthetic IPv6 addresses. F5 Next Policy Enforcer CNF, which provides traffic classification, steering and shaping, and TCP and video optimization. This product is launched as Early Access in February 2023 with basic functionalities. Static TCP optimization is now GA in the initial release. Although the CGNAT (Carrier Grade NAT) and the Policy Enforcer functionalities are specific to User Plane use cases, the Edge Firewall and DNS functionalities have additional uses in other places of the network. F5 and OpenShift BIG-IP Next CNF Solutions fully supports Red Hat OpenShift Container Platform which allows the deployment in edge or core locations with a unified management across the multiple deployments. OpenShift operators greatly facilitates the setup and tuning of telco grade applications. These are: Node Tuning Operator, used to setup Hugepages. CPU Manager and Topology Manager with NUMA awareness which allows to schedule the data plane PODs within a NUMA domain which is aligned with the SR-IOV NICs they are attached to. In an OpenShift platform all these are setup transparently to the applications and BIG-IP Next CNF Solutions uniquely require to be configured with an appropriate runtimeClass. F5 BIG-IP Next CNF Solutions architecture F5 BIG-IP Next CNF Solutions makes use of the widely trusted F5 BIG-IP Traffic Management Microkernel (TMM) data plane. This allows for a high performance, dependable product from the start. The CNF functionalities come from a microservices re-architecture of the broadly used F5 BIG-IP VNFs. The below diagram illustrates how a microservices architecture used. The data plane POD scales vertically from 1 to 16 cores and scales horizontally from 1 to 32 PODs, enabling it to handle millions of subscribers. NUMA nodes are supported. The next diagram focuses on the data plane handling which is the most relevant aspect for this CNF suite: Typically, each data plane POD has two IP address, one for each side of the N6 reference point. These could be named radio and Internet sides as shown in the diagram above. The left-side L3 hop must distribute the traffic amongst the lef-side addresses of the CNF data plane. This left-side L3 hop can be a router with BGP ECMP (Equal Cost Multi Path), an SDN or any other mechanism which is able to: Distribute the subscribers across the data plane PODs, shown in [1] of the figure above. Keep these subscribers in the same PODs when there is a change in the number of active data plane PODs (scale-in, scale-out, maintenance, etc...) as shown in [2] in the figure above. This minimizes service disruption. In the right side of the CNFs, the path towards the Internet, it is typical to implement NAT functionality to transform telco's private addresses to public addresses. This is done with the BIG-IP Next CG-NAT CNF. This NAT makes the return traffic symmetrical by reaching the same POD which processed the outbound traffic. This is thanks to each POD owning part of this NAT space, as shown in [3] of the above figure. Each POD´s NAT address space can be advertised via BGP. When not using NAT in the right side of the CNFs, it is required that the network is able to send the return traffic back to the same POD which is processing the same connection. The traffic must be kept symmetrical at all times, this is typically done with an SDN. Using F5 BIG-IP Next CNF Solutions As expected in a fully integrated Kubernetes solution, both the installation and configuration is done using the Kubernetes APIs. The installation is performed using helm charts, and the configuration using Custom Resource Definitions (CRDs). Unlike using ConfigMaps, using CRDs allow for schema validation of the configurations before these are applied. Details of the CRDs can be found in this clouddocs site. Next it is shown an overview of the most relevant CRDs. General network configuration Deploying in Kubernetes automatically configures and assigns IP addresses to the CNF PODs. The data plane interfaces will require specific configuration. The required steps are: Create Kubernetes NetworkNodePolicies and NetworkAttchment definitions which will allow to expose SR-IOV VFs to the CNF data planes PODs (TMM). To make use of these SR-IOV VFs these are referenced in the BIG-IP controller's Helm chart values file. This is described in the Networking Overview page. Define the L2 and L3 configuration of the exposed SR-IOV interfaces using the F5BigNetVlan CRD. If static routes need to be configured, these can be added using the F5BigNetStaticroute CRD. If BGP configuration needs to be added, this is configured in the BIG-IP controller's Helm chart values file. This is described in the BGP Overview page. It is expected this will be configured using a CRD in the future. Traffic management listener configuration As with classic BIG-IP, once the CNFs are running and plumbed in the network, no traffic is processed by default. The traffic management functionalities implemented by BIG-IP Next CNF Solutions are the same of the analogous modules in the classic BIG-IP, and the CRDs in BIG-IP Next to configure these functionalities are conceptually similar too. Analogous to Virtual Servers in classic BIG-IP, BIG-IP Next CNF Solutions have a set of CRDs that create listeners of traffic where traffic management policies are applied. This is mainly the F5BigContextSecure CRD which allows to specify traffic selectors indicating VLANs, source, destination prefixes and ports where we want the policies to be applied. There are specific CRDs for listeners of Application Level Gateways (ALGs) and protocol specific solutions. These required several steps in classic BIG-IP: first creating the Virtual Service, then creating the profile and finally applying it to the Virtual Server. In BIG-IP Next this is done in a single CRD. At time of this writing, these CRDs are: F5BigZeroratingPolicy - Part of Zero-Rating DNS solution; enabling subscribers to bypass rate limits. F5BigDnsApp - High-performance DNS resolution, caching, and DNS64 translations. F5BigAlgFtp - File Transfer Protocol (FTP) application layer gateway services. F5BigAlgTftp - Trivial File Transfer Protocol (TFTP) application layer gateway services. F5BigAlgPptp - Point-to-Point Tunnelling Protocol (PPTP) application layer gateway services. F5BigAlgRtsp - Real Time Streaming Protocol (RTSP) application layer gateway services. Traffic management profiles and policies configuration Depending on the type of listener created, these can have attached different types of profiles and policies. In the case of F5BigContextSecure it can get attached the following CRDs to define how traffic is processed: F5BigTcpSetting - TCP options to fine-tune how application traffic is managed. F5BigUdpSetting - UDP options to fine-tune how application traffic is managed. F5BigFastl4Setting - FastL4 option to fine-tune how application traffic is managed. and the following policies for security and NAT: F5BigDdosPolicy - Denial of Service (DoS/DDoS) event detection and mitigation. F5BigFwPolicy - Granular stateful-flow filtering based on access control list (ACL) policies. F5BigIpsPolicy - Intelligent packet inspection protects applications from malignant network traffic. F5BigNatPolicy - Carrier-grade NAT (CG-NAT) using large-scale NAT (LSN) pools. The ALG listeners require the use of F5BigNatPolicy and might make use for the F5BigFwPolicy CRDs. These CRDs have also traffic selectors to allow further control over which traffic these policies should be applied to. Firewall Contexts Firewall policies are applied to the listener with best match. In addition to the F5BigFwPolicy that might be attached, a global firewall policy (hence effective in all listeners) can be configured before the listener specific firewall policy is evaluated. This is done with F5BigContextGlobal CRD, which can have attached a F5BigFwPolicy. F5BigContextGlobal also contains the default action to apply on traffic not matching any firewall rule in any context (e.g. Global Context or Secure Context or another listener). This default action can be set to accept, reject or drop and whether to log this default action. In summary, within a listener match, the firewall contexts are processed in this order: ContextGlobal Matching ContextSecure or another listener context. Default action as defined by ContextGlobal's default action. Event Logging Event logging at high speed is critical to provide visibility of what the CNFs are doing. For this the next CRDs are implemented: F5BigLogProfile - Specifies subscriber connection information sent to remote logging servers. F5BigLogHslpub - Defines remote logging server endpoints for the F5BigLogProfile. Demo F5 BIG-IP Next CNF Solutions roadmap What it is being exposed here is just the begin of a journey. Telcos have embraced Kubernetes as compute and orchestration layer. Because of this, BIG-IP Next CNF Solutions will eventually replace the analogous classic BIG-IP VNFs. Expect in the upcoming months that BIG-IP Next CNF Solutions will match and eventually surpass the features currently being offered by the analogous VNFs. Conclusion This article introduces fully re-architected, scalable solution for Red Hat OpenShift mainly focused on telco's user plane. This new microservices architecture offers flexibility, faster service delivery, resiliency and crucially the use of Kubernetes. Kubernetes is becoming the unified orchestration layer for telcos, simplifying infrastructure lifecycle, and reducing costs. OpenShift represents the best-in-class Kubernetes platform thanks to its enterprise readiness and Telco specific features. The architecture of this solution alongside the use of OpenShift also extends network services use cases to the edge by allowing the deployment of Network Functions in a smaller footprint. Please check the official BIG-IP Next CNF Solutions documentation for more technical details and check www.f5.com for a high level overview.
Ulises_Alonso
Feb 20, 2025 Place Technical Articles
2.3KViews
3likes
3Comments
Securing Model Serving in Red Hat OpenShift AI (on ROSA) with F5 Distributed Cloud API Security
Learn how Red Hat OpenShift AI on ROSA and F5 Distributed Cloud API Security work together to protect generative AI model inference endpoints. This integration ensures robust API discovery, schema enforcement, LLM-aware threat detection, bot mitigation, sensitive data redaction, and continuous observability—enabling secure, compliant, and high-performance AI-driven experiences at scale.
Eric_Ji
Jan 03, 2025 Place Technical Articles
778Views
5likes
4Comments
Maximizing Scalability with F5 CIS: Blue-Green Deployment on OpenShift
The following video tutorials and Github repo will help you learn how F5 Container Ingress Services (CIS) can integrate with Red Hat Advanced Cluster Management (ACM) to maximize scalability and provide a per-application blue-green strategy in an OpenShift multi-cluster environment. In this 20 Minute demo you will get a detailed understanding of: Redhat ACM running in a management cluster fetching KubeConfigs from the ConfigMap for worker clusters (Coffee Pod & Tea Pod) F5 CIS subscribing to KubeConfig and listening for events from worker clusters (Coffee Pod & Tea Pod) F5 CIS posting AS3 declaration to BIG-IP. BIG-IP executing all necessary commands to achieve desired end state (ie increase cluster resources to 6 instances per pod) Demo on YouTube video GitHub Repo repo Notes If you want to make sure the service doesn’t take any share of the overall traffic distribution, set the Route base service weight to “0”. Although the pool for the base service gets created (without any pool members for this case), the traffic is distributed among the extended services only. One more point to note is that in the extendedConfigmap mode is set to “ratio”. This is because currently CIS processes the weights only in two cases The route is an A/B route mode is set to ratio However, the cluster ratio doesn't play any role here as we have not defined any cluster ratio values for any of the clusters. So, each cluster will have equal weightage. Therefore, in this case, traffic distribution will be solely based on service weights. (which is the requirement for this case)
Ivan_Hurtt
May 06, 2024 Place Technical Articles
228Views
1like
0Comments
F5 BIG-IP deployment with OpenShift - per-application 2-tier deployments
Introduction This article dives deeper in 2-tier deployments introduced in the previous article F5 BIG-IP deployment with OpenShift - platform and networking options yet we will start comparing a common Kubernetes/OpenShift architecture with a more capable and flexible architecture using BIG-IP. Although this type of deployment can be done with the BIG-IP in Layer 4 (L4) mode this article will focus in using the BIG-IP in Layer 7 (L7) mode which allows for more functionality and flexibility. A regular Kubernetes/OpenShift deployment is as follows: Typically, the External LB doesn´t add much added value and just spreads the connections across the ingress controller instances without any service intelligence. This article emphasizes on the added value of F5 BIG-IP with Container Ingress Services (CIS) by operating at L7. Additionally, it is described how CIS is ingress agnostic and can be used at is fullest regardless of the ingress controller, API manager, service mesh used, or even a combination of these! This 2-tier architecture can be seen next, where it is not only shown that traffic is sent directly to the ingress controllers without the NodePort indirection. In the figure, it can also be seen compared to a 1-tier architecture: ClusterIP and NodePort When using a 2-tier deployment, a key aspect that needs to be decided is how it is going to be sent the traffic to the ingress controllers. NodePort With other External LB solutions (ELB), the common way to send traffic to the ingress controllers in Kubernetes is by exposing these as NodePort. This option always works because the ELB doesn't need to deal with the CNI. In this case, when the ELB sends traffic to a node, the NodePort construct by default load balances the traffic to any node in the cluster where the ingress controller is. This means that there would be 3 hops before reaching the workload POD. These hops are: the ELB, the ingress controller, the NodePort, and finally the workload POD. This NodePort load balancing to other nodes can be removed by changing the externalTrafficPolicy from Cluster to Local in the Service definition. Thus reducing the number of hops from 3 to 2. ClusterIP Alternatively, the BIG-IP can send the traffic directly to the POD addresses, aka ClusterIP mode, without any translation or Kubernetes construct in between the BIG-IP and the ingress controller. This is specified in CIS with the --pool-member-type=cluster option and requires support of the CNI. Using OpenShiftSDN and ClusterIP is discouraged because this CNI is being deprecated and will not be covered in this article. When using OVNKubernetes, the BIG-IP sends the traffic without any tunnelling. At this moment, this requires that the clusters are in the same subnet as the BIG-IP. Which option to choose? If the OpenShift cluster is not in the same subnet as the BIG-IP or if the infrastructure doesn´t allow POD addresses traverse the segment (ie: because of a L2 firewall/micro-segmentation), then NodePort would need to be used. I would discourage the use of NodePort with externalTrafficPolicy: Cluster because it doesn´t allow to do adequate health check monitoring because each probe will be sent to a different ingress controller. Moreover, persistence and troubleshooting is more tricky too because of the additional hop that NodePort creates. If NodePort needs to be used, then externalTrafficPolicy: Local referencing the ingress controllers is recommended. If there is no problem in routing the traffic to the OpenShift nodes using the POD addresses, I would encourage using ClusterIP because it is a transparent method, where the application traffic is easier to track: there are no additional hops or address translations. OpenShift´s bundled ingress/router is HA-proxy; In on-prem OpenShift deployments, the default router instance is created using hostNetwork access. HostNetwork can also be used with CIS --pool-member-type=cluster mode. The IP addresses of the pool members will be the node addresses, but there will not be any indirection to the POD. The examples provided in the git repository mentioned in this article use ClusterIP but can be easily modified to NodePort if necessary. 2-tier per-service load balancing and health checking This type of deployment requires that the external load balancer, in this case the F5 BIG-IP, can take actions based on the L7 route path. This includes health-checking the application and performing TLS/SSL termination. From a Kubernetes/OpenShift point of view, the most remarkable is that it is needed to expose the L7 routes twice. That is, for a given L7 route there will be a manifest for the ingress controller and another L7 route manifest for CIS. This is outlined in the next figure: In the above figure, there is only one ingress controller instance with several replicas serving the same L7 routes. It is not relevant which ingress controller it is. This ingress controller can have defined its L7 routes with any resource type. This is transparent from the point of view of CIS. CIS only requires that the ingress controller can be referenced with a selector from a Service. This service is shown in brown. There will be a Service manifest for each L7 route for CIS. Even if all these services point to the same ingress controller instances. This allows you to: Monitor each L7 route´s backends individually. Have a clean NetOps/DevOps separation where DevOps can manipulate the ingress controller configurations freely and NetOps can control the publishing of these in the ELB. Ultimately make load-balancing decisions in a per-application basis The L7 routes for CIS can be defined either as VirtualServer, Route (as NextGen Route) or as Ingress resource types. Using the Ingress resource type is not recommended because of its limited functionalities compared with the other types. Overall, there will be a 1:1:1 mapping between the L7 routes defined for CIS, the Service manifests referenced by CIS´ L7 routes, and the L7 routes defined for the ingress controller. This is shown conceptually in the next figure. With respect to load balancing algorithm, given that the number of ingress controllers in a two-tier deployment is typically small it is sensible to take into account the load of each ingress controller by using an algorithm such as least-connections, least-sessions (if using persistence) or fastest (application response time) which should send more traffic to the ingress controller which is performing better or in other words, would avoid sending more traffic to an ingress controller that is not performing well. Unify all ingresses in a single VIP Thanks to CIS being ingress controller agnostic, it is also possible to combine L7 routes from different ingress controllers, these can be a mix of one or more ingress controllers, including service mesh ingress or API managers. This brings the flexibility of exposing L7 routes from these sources in a single VIP, as shown in the next figure. From the picture above, it can be seen how in this arrangement the ingress controllers are not limited to a single L7 route. Also note that as mentioned previously, it is needed to have a 1:1 mapping between the L7 routes defined in the BIG-IP (1st tier) and in the in-cluster ingress element (2nd tier). By default, these paired L7 routes are expected to match their URL. If necessary, it is possible to do URL translation by using the hostRewrite and rewrite attributes in the VirtualServer CR. OpenShift´s documentation mentions several use cases for creating additional OpenShift router (HA-proxy) instances, referred to as route sharding. These are: Balance Ingress Controllers, or routers, with several routes to speed up responses to changes. Allocate certain routes to have different reliability guarantees than other routes. Allow certain Ingress Controllers to have different policies defined. Allow only specific routes to use additional features. Expose different routes on different addresses so that internal and external users can see different routes, for example. Transfer traffic from one version of an application to another during a blue-green deployment. An additional use case is to isolate the control-plane Routes of the cluster and the application Routes in independent HA-proxy instances. This way it is not only it can be guaranteed different levels of reliability but also guaranteeing resource allocation for the cluster´s own traffic. In the section "Using additional Router instances" section below, it is shown how to do these configurations. Monitoring in a 2-tier deployment For each L7 route, regardless of the resource type used (Ingress, Route or VirtualServer), these share the same schema for monitoring. Next are shown the relevant parameters, where send is the key parameter that specifies L7 route: monitor: type: http send: "GET / HTTP/1.1\r\nHost: www.twotier.com\r\nConnection: close\r\n\r\n" recv: "^HTTP/1.1 200" In many cases, https will be used instead. HA-proxy requires that the health monitor uses TLS 1.2. By default, the https monitor defaults to TLS 1.0, hence we will need to specify one of the BIG-IP default TLS profiles as shown next: monitor: type: https send: "GET / HTTP/1.1\r\nHost: www.twotier.com\r\nConnection: close\r\n\r\n" recv: "^HTTP/1.1 200" sslProfile: /Common/serverssl When using NGINX or Istio ingress gateway, using TLS SNI (Server Name Indication) is requierd. In cases where SNI is required, the following needs to be done: One time, in TMM monitoring needs to be enabled as indicated in https://my.f5.com/manage/s/article/K85210713 For each FQDN it is needed to pre-create a server-side TLS profile in the partition /Common specifying the server name used as TLS SNI. The resulting monitor section will typically look as follows: monitor: type: https send: "GET / HTTP/1.1\r\nHost: www.twotier.com\r\nConnection: close\r\n\r\n" recv: "^HTTP/1.1 200" sslProfile: /Common/serverssl-www.twotier.com Note that this TLS profile is only needed for the monitoring, for application´s traffic the BIG-IP forwards the SNI received in the VIP. Alternative monitoring in a 2-tier deployment In this article, it is preferred to have 1:1 mapping between the L7 routes defined in the BIG-IP and the L7 routes defined in the ingress controller. It is possible to reduce the number of L7 routes and Service manifests for CIS by adding several monitors to a single L7 route manifest. For example we could have a single base L7 route such as www.twotier.com/ and several monitors probing the different applications behind this FQDN, for example: www.twotier.com/, www.twotier.com/account, and www.twotier.com/shop. This alternative monitoring can be configured with the following parameters: In this example, you can see how to create such a configuration. The preference for using a 1:1 mapping instead is because although this alternative strategy creates less manifests, automating the creation of these might be more complex. Additionally, it will be less easy to know in a given moment which applications are not working fine. Using OpenShift´s Router (HA-Proxy) in the second tier CIS can be configured to ingest F5 CRs, OpenShift Routes CRs or Ingress resources. When CIS is configured for using Routes or Ingress, by default HA-Proxy will also ingest these. In other words, both CIS and HA-Proxy will implement the same L7 routes. This behavior also applies to single-tier deployments where the behavior could be desired for testing. In general this behavior is not desired. HA-Proxy can be configured to do-not evaluate Routes or Ingress manifests with a given label or to do-not search for these in namespaces with a given label, for example "router=bigip". This behavior is configured in the IngressController resource type that manages HA-proxy's configuration: { "spec": { "namespaceSelector": { "matchExpressions": [ { "key": "router", "operator": "NotIn", "values": [ "bigip" ] } ] } } } This JSON configuration is applied with the next command for the default ingress controller/router in OpenShift: oc -n openshift-ingress-operator patch ingresscontroller default --patch-file=router-default.shard.json --type=merge More information on Route sharding can be found in this OpenShift document. Using the default Router instance OpenShift´s installer deploys the default HA-proxy instance with endpointPublishingStrategy type set to either hostNetwork or LoadBalancer. This will depend whether the OpenShift deployment is on-prem, in a public cloud, etc... This can be checked with the following command: oc get ingresscontrollers default -n openshift-ingress-operator -o jsonpath='{.status.endpointPublishingStrategy}{"\n"}' Regardless HA-proxy is configured with endpointPublishingStrategy either hostNetwork or LoadBalancer, CIS can be configured in clusterIP mode and refer to the HA-Proxy PODs using the "--orchestration-cni=ovn-k8s" and "--pool-member-type=cluster" options. The Service to refer to these HA-Proxy PODs will look as follows: apiVersion: v1 kind: Service metadata: name: router-default-route-a namespace: openshift-ingress spec: ports: - name: http port: 80 protocol: TCP targetPort: http - name: https port: 443 protocol: TCP targetPort: https selector: ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default type: ClusterIP Where as indicated earlier there will be a Service manifest for each L7 route in order to have per L7 route monitoring. Note that these Services reside in the openshift-ingress namespace, where the HA-Proxy PODs are. Examples of this configuration can be found here using F5 CRs. Using additional Router instances Creating a new HA-Proxy instance just requires creating a new IngressController manifest in the openshift-ingress-operator namespace, such as the following sample: apiVersion: operator.openshift.io/v1 kind: IngressController metadata: name: shard-apps namespace: openshift-ingress-operator spec: clientTLS: defaultCertificate: name: router-shard-secret domain: shard-apps.com endpointPublishingStrategy: type: Private namespaceSelector: matchExpressions: - key: router operator: In values: - shard-apps replicas: 1 From this sample, note that: This new Router instance will evaluate only the L7 routes with namespaces with the label "router=shard-apps". the endpointPublishingStrategy is set to type Private, this is because we don´t need further exposure of it when CIS is accessing it in ClusterIP mode. When using an additional HA-proxy instance, the default Router will need to be instructed not to evaluate these same L7 routes as the additional HA-proxy instance. This is shown next: { "spec": { "namespaceSelector": { "matchExpressions": [ { "key": "router", "operator": "NotIn", "values": [ "bigip", "shard-apps" ] } ] } } } An example of this configuration can be found here using Routes and multiple shards. Using Istio Ingress Gateway / OpenShift Service Mesh Using Istio (aka Service Mesh as packaged in OpenShift) in the second tier is straight-forward because Istio and CIS use different resource types for configuring L7 routes. A sample Service manifest to point to Istio ingress gateway is shown next apiVersion: v1 kind: Service metadata: name: svc-route-a namespace: istio-system spec: ports: - name: https port: 8443 protocol: TCP targetPort: https selector: app: istio-ingressgateway type: ClusterIP Where it can be observed that the Service manifests need to be created in the istio-system namespace and the selector needs to match the Istio ingress gateway´s Deployment's label. An example of this configuration can be found here using F5 CRs. Using NGINX Using NGINX in the second tier is much like using Istio, next is shown a sample Service manifest pointing to NGINX instances: apiVersion: v1 kind: Service metadata: name: svc-route-a namespace: nginx-ingress spec: ports: - name: https port: 443 protocol: TCP targetPort: 443 selector: app.kubernetes.io/instance: nginxingress-sample app.kubernetes.io/name: nginx-ingress type: ClusterIP The only special consideration to be considered, and only if both NGINX and CIS are meant to use the Ingress resource type, is to specify in the Ingress manifests which controller will ingest the manifest. This is specified with the ingressClassName attribute as shown next: in the case of NGINX apiVersion: networking.k8s.io/v1 kind: Ingress [...] spec: ingressClassName: nginx or in the case of CIS spec: ingressClassName: f5 An example of this configuration using F5 CRs in CIS and Ingress in NGINX can be found Using advanced BIG-IP services with CIS BIG-IP has many advanced services that can be easily referenced using F5 VirtualServer CRs, either directly in the manifest or referencing a Policy CR. When using Next Gen Routes these can also reference Policy CR but in a per router-group basis, hence having somewhat less granularity. Next is an overview of some of these advanced functionalities available. VirtualServer CR attributes for configuring advanced services Functionality attribute Advanced Web Application Firewall waf L3 anti-DoS protection dos L7 bot defense botDefense HTML content rewrite htmlProfile Custom iRules to perform advanced traffic management iRules Message routing framework (MRF) functionality for non HTTP traffic httpMrfRoutingEnabled IP intelligence ipIntelligencePolicy L3 Firewall rules firewallPolicy The configuration of these services must be pre-created in the BIG-IP as profiles in the /Common partition which can be referenced freely by CIS. It is expected that CIS 2.17 will incorporate APM and SSLo. Conclusion and closing remarks F5 BIG-IP is an External Loadbalancer with application awareness which allows to do advanced L7 services (not just TLS termination) and unify the different ingress paths into the cluster, allowing to merge several ingress controllers, API managers, service mesh or a combination of these. This solution is ingress type agnostic. A two-tier arrangement allows clean DevOps and NetOps separation. This is done by having separate L7 routes defined for the in-cluster ingress controllers and for the BIG-IPs. To continue your journey, please check the examples in this GitHub repository using VirtualServer F5 CRs and OpenShift Route CRs.
Ulises_Alonso
Apr 22, 2024 Place Technical Articles
902Views
0likes
0Comments
F5 BIG-IP per application Red Hat OpenShift cluster migrations
Overview OpenShift migrations are typically done when it is desired to minimise disruption time when performing cluster upgrades. Disruptions can especially occur when performing big changes in the cluster such as changing the CNI from OpenShiftSDN to OVNKubernetes. OpenShift cluster migrations are well covered for applications by using RedHat's Migration Toolkit for Containers (MTC). The F5 BIG-IP has the role of network redirector indicated in the Network considerations chapter. The F5 BIG-IP can perform per L7 route migration without service disruption, hence allowing migration or roll-back on a per-application basis, eliminating disruption and de-risking the maintenance window. How it works As mentioned above, the traffic redirection will be done on a per L7 route basis, this is true regardless of how these L7 routes are implemented: ingress controller, API manager, service mesh, or a combination of these. This L7 awareness is achieved by using F5 BIG-IP's Controller Ingress Services (CIS) controller for Kubernetes/OpenShift and its multi-cluster functionality which can expose in a single VIP L7 routes of services hosted in multiple Kubernetes/OpenShift clusters. This is shown in the next picture. For a migration operation it will be used a blue/green strategy independent for each L7 route where blue will refer to the application in the older cluster and green will refer to the application in the newer cluster. For each L7 route, it will be specified a weight for each blue or green backend (like in an A/B strategy). This is shown in the next picture. In this example, the migration scenario uses OpenShift´s default ingress controller (HA proxy) as an in-cluster ingress controller where the Route CR is used to indicate the L7 routes. For each L7 route defined in the HA-proxy tier, it will be defined as an L7 route in the F5 BIG-IP tier. This 1:1 mapping allows to have the per-application granularity. The VirtualServer CR is used for the F5 BIG-IP. If desired, it is also possible to use Route resources for the F5 BIG-IP. Next, it is shown the manifests for a given L7 route required for the F5 BIG-IP, in this case, https://www.migration.com/shop (alias route-b) apiVersion: "cis.f5.com/v1" kind: VirtualServer metadata: name: route-b namespace: openshift-ingress labels: f5cr: "true" spec: host: www.migration.com virtualServerAddress: "10.1.10.106" hostGroup: migration.com tlsProfileName: reencrypt-tls profileMultiplex: "/Common/oneconnect-32" pools: - path: /shop service: router-default-route-b-ocp1 servicePort: 443 weight: 100 alternateBackends: - service: router-default-route-b-ocp2 weight: 0 monitor: type: https name: /Common/www.migration.com-shop reference: bigip --- apiVersion: v1 kind: Service metadata: annotations: name: router-default-route-b-ocp1 namespace: openshift-ingress spec: ports: - name: http port: 80 protocol: TCP targetPort: http - name: https port: 443 protocol: TCP targetPort: https selector: ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default type: NodePort --- apiVersion: v1 kind: Service metadata: annotations: name: router-default-route-b-ocp2 namespace: openshift-ingress spec: ports: - name: http port: 80 protocol: TCP targetPort: http - name: https port: 443 protocol: TCP targetPort: https selector: ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default type: NodePort The CIS multi-cluster feature will search for the specified services in both clusters. It is up to DevOps to ensure that blue services are only present in the cluster designated as blue (in this case router-default-route-b-ocp1) and green services are only present in the cluster designated as green (in this case router-default-route-b-ocp2). It is important to remark that the Route manifests for HA-proxy (or any other ingress solution used) doesn't require any modification. That is, this migration mechanism is transparent to the application developers. Demo You can see this feature in action in the next video The manifests used in the demo can be found in the following GitHub repository: https://github.com/f5devcentral/f5-bd-cis-demo/tree/main/crds/demo-mc-twotier-haproxy-noshards Next steps Try it today! CIS is open-source software and is included in your support entitlement. If you want to learn more about CIS and CIS multi-cluster features the following blog articles are suggested. F5 BIG-IP deployment with OpenShift - platform and networking options F5 BIG-IP deployment with OpenShift - publishing application options F5 BIG-IP deployment with OpenShift - multi-cluster architectures
Ulises_Alonso
Mar 28, 2024 Place Technical Articles
491Views
0likes
0Comments
F5 BIG-IP deployment with Red Hat OpenShift - the no CIS option
This article is a set of recommendations when using BIG-IP as external load balancer of an OpenShift cluster with the default router (HA-proxy) and not using CIS
Ulises_Alonso
Mar 25, 2024 Place Technical Articles
1.7KViews
0likes
0Comments
F5 BIG-IP deployment with OpenShift - multi-cluster architectures
This functionality enables to load balance services that are spread across multiple clusters. It is important to remark that this functionality is Service oriented: the load balancing decisions are independent for each Service.
Ulises_Alonso
Feb 20, 2024 Place Technical Articles
1.2KViews
0likes
0Comments
Scale Multi-Cluster OpenShift Deployments with F5 Container Ingress Services
In the realm of enterprise IT, the complexity of managing applications across multiple clusters has long been a significant challenge. Recognizing this, we have introduced a solution with the latest release of BIG-IP Container Ingress Service (CIS). This article explores the challenges of distributing traffic across multiple clusters and how F5's CIS addresses these issues head-on. The Importance of Distributing Traffic Across Multiple Clusters Distributing traffic across multiple clusters is essential for maintaining high availability, ensuring consistent performance, and facilitating agile deployment strategies. However, achieving this is challenged by the complexity of configuring and managing multiple clusters, ensuring consistent application policies, and maintaining uptime during upgrades or deployments. Introducing the Solution: F5 BIG-IP CIS with Multi-Cluster Support F5's BIG-IP CIS now offers multi-cluster support, revolutionizing how applications are deployed across multiple OpenShift clusters. This new feature caters to a variety of deployment strategies and operational requirements: Deployment Strategies: Supports both Blue-Green Deployments and A/B Testing, allowing for more flexible and risk-averse application rollouts. High Availability (HA): Ensures continuous application availability by spanning across 'N' clusters, reducing the risk of downtime. Load Balancing: Implements Ratio-Based Load Distribution, optimizing resource utilization and ensuring efficient traffic management. Cluster Management: Facilitates Seamless Cluster Upgrades, ensuring that the latest features and security updates are implemented without impacting ongoing operations. How F5 BIG-IP CIS Works The picture below shows how the Multi-Cluster feature works. Notice that in the picture, we are showing 2 OpenShift Cluster, each with a CIS instance running. In a real deployment, we can run more than 2 clusters, but with only 2 CIS instances running in 2 of the clusters for redundancy purposes. Here is a brief workflow of the solution: Fetch KubeConfigs from ConfigMap: This step involves accessing the ConfigMap repository and extracting the necessary KubeConfig files for each external cluster. Subscribes to Informer Events: Utilizes the previously fetched KubeConfig to set up informers for external clusters. The informers continuously listen for and capture any events or changes occurring in these clusters, thereby maintaining a real-time monitoring system. Heartbeat Check for Primary CIS: The workflow includes regular heartbeat checks on the primary CIS. This involves sending periodic signals to the primary CIS to verify its operational status and ensure it is actively functioning. Post AS3 Declaration: In this final step, the system coordinates the posting of AS3 declarations to the external clusters. This is managed to ensure that only one CIS posts an AS3 declaration at any given time, thereby avoiding conflicts and ensuring orderly updates. Conclusion F5's BIG-IP CIS with Multi-Cluster Support is a strategic enabler for businesses relying on OpenShift. By simplifying the management of multi-cluster environments, organizations can focus more on innovation and less on multi-cluster application operational complexities. Watch our detailed demo on the F5 DevCentral YouTube channel to see how this solution can transform your OpenShift deployments. Explore the Demo:
Eric_Ji
Jan 17, 2024 Place Technical Articles
290Views
2likes
0Comments

"}},"componentScriptGroups({\"componentId\":\"custom.widget.Beta_MetaNav\"})":{"__typename":"ComponentScriptGroups","scriptGroups":{"__typename":"ComponentScriptGroupsDefinition","afterInteractive":{"__typename":"PageScriptGroupDefinition","group":"AFTER_INTERACTIVE","scriptIds":[]},"lazyOnLoad":{"__typename":"PageScriptGroupDefinition","group":"LAZY_ON_LOAD","scriptIds":[]}},"componentScripts":[]},"component({\"componentId\":\"custom.widget.Beta_Footer\"})":{"__typename":"Component","render({\"context\":{\"component\":{\"entities\":[],\"props\":{}},\"page\":{\"entities\":[],\"name\":\"TagPage\",\"props\":{},\"url\":\"https://community.f5.com/tag/Red%20Hat%20OpenShift\"}}})":{"__typename":"ComponentRenderResult","html":"

Trademarks Policies Privacy California Privacy Do Not Sell My Personal Information

"}},"componentScriptGroups({\"componentId\":\"custom.widget.Beta_Footer\"})":{"__typename":"ComponentScriptGroups","scriptGroups":{"__typename":"ComponentScriptGroupsDefinition","afterInteractive":{"__typename":"PageScriptGroupDefinition","group":"AFTER_INTERACTIVE","scriptIds":[]},"lazyOnLoad":{"__typename":"PageScriptGroupDefinition","group":"LAZY_ON_LOAD","scriptIds":[]}},"componentScripts":[]},"component({\"componentId\":\"custom.widget.Tag_Manager_Helper\"})":{"__typename":"Component","render({\"context\":{\"component\":{\"entities\":[],\"props\":{}},\"page\":{\"entities\":[],\"name\":\"TagPage\",\"props\":{},\"url\":\"https://community.f5.com/tag/Red%20Hat%20OpenShift\"}}})":{"__typename":"ComponentRenderResult","html":" "}},"componentScriptGroups({\"componentId\":\"custom.widget.Tag_Manager_Helper\"})":{"__typename":"ComponentScriptGroups","scriptGroups":{"__typename":"ComponentScriptGroupsDefinition","afterInteractive":{"__typename":"PageScriptGroupDefinition","group":"AFTER_INTERACTIVE","scriptIds":[]},"lazyOnLoad":{"__typename":"PageScriptGroupDefinition","group":"LAZY_ON_LOAD","scriptIds":[]}},"componentScripts":[]},"component({\"componentId\":\"custom.widget.Consent_Blackbar\"})":{"__typename":"Component","render({\"context\":{\"component\":{\"entities\":[],\"props\":{}},\"page\":{\"entities\":[],\"name\":\"TagPage\",\"props\":{},\"url\":\"https://community.f5.com/tag/Red%20Hat%20OpenShift\"}}})":{"__typename":"ComponentRenderResult","html":"

"}},"componentScriptGroups({\"componentId\":\"custom.widget.Consent_Blackbar\"})":{"__typename":"ComponentScriptGroups","scriptGroups":{"__typename":"ComponentScriptGroupsDefinition","afterInteractive":{"__typename":"PageScriptGroupDefinition","group":"AFTER_INTERACTIVE","scriptIds":[]},"lazyOnLoad":{"__typename":"PageScriptGroupDefinition","group":"LAZY_ON_LOAD","scriptIds":[]}},"componentScripts":[]},"cachedText({\"lastModified\":\"1743097588266\",\"locale\":\"en-US\",\"namespaces\":[\"components/community/NavbarDropdownToggle\"]})":[{"__ref":"CachedAsset:text:en_US-components/community/NavbarDropdownToggle-1743097588266"}],"cachedText({\"lastModified\":\"1743097588266\",\"locale\":\"en-US\",\"namespaces\":[\"shared/client/components/common/OverflowNav\"]})":[{"__ref":"CachedAsset:text:en_US-shared/client/components/common/OverflowNav-1743097588266"}],"cachedText({\"lastModified\":\"1743097588266\",\"locale\":\"en-US\",\"namespaces\":[\"components/messages/MessageView/MessageViewInline\"]})":[{"__ref":"CachedAsset:text:en_US-components/messages/MessageView/MessageViewInline-1743097588266"}],"cachedText({\"lastModified\":\"1743097588266\",\"locale\":\"en-US\",\"namespaces\":[\"components/customComponent/CustomComponent\"]})":[{"__ref":"CachedAsset:text:en_US-components/customComponent/CustomComponent-1743097588266"}],"cachedText({\"lastModified\":\"1743097588266\",\"locale\":\"en-US\",\"namespaces\":[\"components/users/UserLink\"]})":[{"__ref":"CachedAsset:text:en_US-components/users/UserLink-1743097588266"}],"cachedText({\"lastModified\":\"1743097588266\",\"locale\":\"en-US\",\"namespaces\":[\"components/messages/MessageSubject\"]})":[{"__ref":"CachedAsset:text:en_US-components/messages/MessageSubject-1743097588266"}],"cachedText({\"lastModified\":\"1743097588266\",\"locale\":\"en-US\",\"namespaces\":[\"components/messages/MessageTime\"]})":[{"__ref":"CachedAsset:text:en_US-components/messages/MessageTime-1743097588266"}],"cachedText({\"lastModified\":\"1743097588266\",\"locale\":\"en-US\",\"namespaces\":[\"shared/client/components/nodes/NodeIcon\"]})":[{"__ref":"CachedAsset:text:en_US-shared/client/components/nodes/NodeIcon-1743097588266"}],"cachedText({\"lastModified\":\"1743097588266\",\"locale\":\"en-US\",\"namespaces\":[\"components/messages/MessageUnreadCount\"]})":[{"__ref":"CachedAsset:text:en_US-components/messages/MessageUnreadCount-1743097588266"}],"cachedText({\"lastModified\":\"1743097588266\",\"locale\":\"en-US\",\"namespaces\":[\"components/messages/MessageViewCount\"]})":[{"__ref":"CachedAsset:text:en_US-components/messages/MessageViewCount-1743097588266"}],"cachedText({\"lastModified\":\"1743097588266\",\"locale\":\"en-US\",\"namespaces\":[\"components/kudos/KudosCount\"]})":[{"__ref":"CachedAsset:text:en_US-components/kudos/KudosCount-1743097588266"}],"cachedText({\"lastModified\":\"1743097588266\",\"locale\":\"en-US\",\"namespaces\":[\"components/messages/MessageRepliesCount\"]})":[{"__ref":"CachedAsset:text:en_US-components/messages/MessageRepliesCount-1743097588266"}],"cachedText({\"lastModified\":\"1743097588266\",\"locale\":\"en-US\",\"namespaces\":[\"components/messages/MessageBody\"]})":[{"__ref":"CachedAsset:text:en_US-components/messages/MessageBody-1743097588266"}],"cachedText({\"lastModified\":\"1743097588266\",\"locale\":\"en-US\",\"namespaces\":[\"shared/client/components/users/UserAvatar\"]})":[{"__ref":"CachedAsset:text:en_US-shared/client/components/users/UserAvatar-1743097588266"}]},"CachedAsset:pages-1742462581816":{"__typename":"CachedAsset","id":"pages-1742462581816","value":[{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"HowDoI.GetInvolved.MvpProgram","type":"COMMUNITY","urlPath":"/c/how-do-i/get-involved/mvp-program","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"BlogViewAllPostsPage","type":"BLOG","urlPath":"/category/:categoryId/blog/:boardId/all-posts/(/:after|/:before)?","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"CasePortalPage","type":"CASE_PORTAL","urlPath":"/caseportal","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"CreateGroupHubPage","type":"GROUP_HUB","urlPath":"/groups/create","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"CaseViewPage","type":"CASE_DETAILS","urlPath":"/case/:caseId/:caseNumber","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"InboxPage","type":"COMMUNITY","urlPath":"/inbox","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"HowDoI.GetInvolved.AdvocacyProgram","type":"COMMUNITY","urlPath":"/c/how-do-i/get-involved/advocacy-program","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"HowDoI.GetHelp.NonCustomer","type":"COMMUNITY","urlPath":"/c/how-do-i/get-help/non-customer","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"HelpFAQPage","type":"COMMUNITY","urlPath":"/help","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"HowDoI.GetHelp.F5Customer","type":"COMMUNITY","urlPath":"/c/how-do-i/get-help/f5-customer","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"IdeaMessagePage","type":"IDEA_POST","urlPath":"/idea/:boardId/:messageSubject/:messageId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"IdeaViewAllIdeasPage","type":"IDEA","urlPath":"/category/:categoryId/ideas/:boardId/all-ideas/(/:after|/:before)?","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"LoginPage","type":"USER","urlPath":"/signin","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"BlogPostPage","type":"BLOG","urlPath":"/category/:categoryId/blogs/:boardId/create","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"HowDoI.GetInvolved","type":"COMMUNITY","urlPath":"/c/how-do-i/get-involved","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"HowDoI.Learn","type":"COMMUNITY","urlPath":"/c/how-do-i/learn","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1739501996000,"localOverride":null,"page":{"id":"Test","type":"CUSTOM","urlPath":"/custom-test-2","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"ThemeEditorPage","type":"COMMUNITY","urlPath":"/designer/themes","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"TkbViewAllArticlesPage","type":"TKB","urlPath":"/category/:categoryId/kb/:boardId/all-articles/(/:after|/:before)?","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"OccasionEditPage","type":"EVENT","urlPath":"/event/:boardId/:messageSubject/:messageId/edit","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"OAuthAuthorizationAllowPage","type":"USER","urlPath":"/auth/authorize/allow","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"PageEditorPage","type":"COMMUNITY","urlPath":"/designer/pages","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"PostPage","type":"COMMUNITY","urlPath":"/category/:categoryId/:boardId/create","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"ForumBoardPage","type":"FORUM","urlPath":"/category/:categoryId/discussions/:boardId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"TkbBoardPage","type":"TKB","urlPath":"/category/:categoryId/kb/:boardId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"EventPostPage","type":"EVENT","urlPath":"/category/:categoryId/events/:boardId/create","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"UserBadgesPage","type":"COMMUNITY","urlPath":"/users/:login/:userId/badges","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"GroupHubMembershipAction","type":"GROUP_HUB","urlPath":"/membership/join/:nodeId/:membershipType","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"MaintenancePage","type":"COMMUNITY","urlPath":"/maintenance","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"IdeaReplyPage","type":"IDEA_REPLY","urlPath":"/idea/:boardId/:messageSubject/:messageId/comments/:replyId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"UserSettingsPage","type":"USER","urlPath":"/mysettings/:userSettingsTab","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"GroupHubsPage","type":"GROUP_HUB","urlPath":"/groups","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"ForumPostPage","type":"FORUM","urlPath":"/category/:categoryId/discussions/:boardId/create","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"OccasionRsvpActionPage","type":"OCCASION","urlPath":"/event/:boardId/:messageSubject/:messageId/rsvp/:responseType","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"VerifyUserEmailPage","type":"USER","urlPath":"/verifyemail/:userId/:verifyEmailToken","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"AllOccasionsPage","type":"OCCASION","urlPath":"/category/:categoryId/events/:boardId/all-events/(/:after|/:before)?","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"EventBoardPage","type":"EVENT","urlPath":"/category/:categoryId/events/:boardId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"TkbReplyPage","type":"TKB_REPLY","urlPath":"/kb/:boardId/:messageSubject/:messageId/comments/:replyId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"IdeaBoardPage","type":"IDEA","urlPath":"/category/:categoryId/ideas/:boardId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"CommunityGuideLinesPage","type":"COMMUNITY","urlPath":"/communityguidelines","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"CaseCreatePage","type":"SALESFORCE_CASE_CREATION","urlPath":"/caseportal/create","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"TkbEditPage","type":"TKB","urlPath":"/kb/:boardId/:messageSubject/:messageId/edit","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"ForgotPasswordPage","type":"USER","urlPath":"/forgotpassword","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"IdeaEditPage","type":"IDEA","urlPath":"/idea/:boardId/:messageSubject/:messageId/edit","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"TagPage","type":"COMMUNITY","urlPath":"/tag/:tagName","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"BlogBoardPage","type":"BLOG","urlPath":"/category/:categoryId/blog/:boardId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"OccasionMessagePage","type":"OCCASION_TOPIC","urlPath":"/event/:boardId/:messageSubject/:messageId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"ManageContentPage","type":"COMMUNITY","urlPath":"/managecontent","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"ClosedMembershipNodeNonMembersPage","type":"GROUP_HUB","urlPath":"/closedgroup/:groupHubId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"HowDoI.GetHelp.Community","type":"COMMUNITY","urlPath":"/c/how-do-i/get-help/community","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"CommunityPage","type":"COMMUNITY","urlPath":"/","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"HowDoI.GetInvolved.ContributeCode","type":"COMMUNITY","urlPath":"/c/how-do-i/get-involved/contribute-code","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"ForumMessagePage","type":"FORUM_TOPIC","urlPath":"/discussions/:boardId/:messageSubject/:messageId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"IdeaPostPage","type":"IDEA","urlPath":"/category/:categoryId/ideas/:boardId/create","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"BlogMessagePage","type":"BLOG_ARTICLE","urlPath":"/blog/:boardId/:messageSubject/:messageId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"RegistrationPage","type":"USER","urlPath":"/register","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"EditGroupHubPage","type":"GROUP_HUB","urlPath":"/group/:groupHubId/edit","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"ForumEditPage","type":"FORUM","urlPath":"/discussions/:boardId/:messageSubject/:messageId/edit","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"ResetPasswordPage","type":"USER","urlPath":"/resetpassword/:userId/:resetPasswordToken","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"TkbMessagePage","type":"TKB_ARTICLE","urlPath":"/kb/:boardId/:messageSubject/:messageId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"HowDoI.Learn.AboutIrules","type":"COMMUNITY","urlPath":"/c/how-do-i/learn/about-irules","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"BlogEditPage","type":"BLOG","urlPath":"/blog/:boardId/:messageSubject/:messageId/edit","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"HowDoI.GetHelp.F5Support","type":"COMMUNITY","urlPath":"/c/how-do-i/get-help/f5-support","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"ManageUsersPage","type":"USER","urlPath":"/users/manage/:tab?/:manageUsersTab?","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"ForumReplyPage","type":"FORUM_REPLY","urlPath":"/discussions/:boardId/:messageSubject/:messageId/replies/:replyId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"PrivacyPolicyPage","type":"COMMUNITY","urlPath":"/privacypolicy","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"NotificationPage","type":"COMMUNITY","urlPath":"/notifications","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"UserPage","type":"USER","urlPath":"/users/:login/:userId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"HealthCheckPage","type":"COMMUNITY","urlPath":"/health","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"OccasionReplyPage","type":"OCCASION_REPLY","urlPath":"/event/:boardId/:messageSubject/:messageId/comments/:replyId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"ManageMembersPage","type":"GROUP_HUB","urlPath":"/group/:groupHubId/manage/:tab?","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"SearchResultsPage","type":"COMMUNITY","urlPath":"/search","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"BlogReplyPage","type":"BLOG_REPLY","urlPath":"/blog/:boardId/:messageSubject/:messageId/replies/:replyId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"GroupHubPage","type":"GROUP_HUB","urlPath":"/group/:groupHubId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"TermsOfServicePage","type":"COMMUNITY","urlPath":"/termsofservice","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"HowDoI.GetHelp","type":"COMMUNITY","urlPath":"/c/how-do-i/get-help","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"HowDoI.GetHelp.SecurityIncident","type":"COMMUNITY","urlPath":"/c/how-do-i/get-help/security-incident","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"CategoryPage","type":"CATEGORY","urlPath":"/category/:categoryId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"ForumViewAllTopicsPage","type":"FORUM","urlPath":"/category/:categoryId/discussions/:boardId/all-topics/(/:after|/:before)?","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"TkbPostPage","type":"TKB","urlPath":"/category/:categoryId/kbs/:boardId/create","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"GroupHubPostPage","type":"GROUP_HUB","urlPath":"/group/:groupHubId/:boardId/create","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742462581816,"localOverride":null,"page":{"id":"HowDoI","type":"COMMUNITY","urlPath":"/c/how-do-i","__typename":"PageDescriptor"},"__typename":"PageResource"}],"localOverride":false},"CachedAsset:text:en_US-components/context/AppContext/AppContextProvider-0":{"__typename":"CachedAsset","id":"text:en_US-components/context/AppContext/AppContextProvider-0","value":{"noCommunity":"Cannot find community","noUser":"Cannot find current user","noNode":"Cannot find node with id {nodeId}","noMessage":"Cannot find message with id {messageId}"},"localOverride":false},"CachedAsset:text:en_US-shared/client/components/common/Loading/LoadingDot-0":{"__typename":"CachedAsset","id":"text:en_US-shared/client/components/common/Loading/LoadingDot-0","value":{"title":"Loading..."},"localOverride":false},"User:user:-1":{"__typename":"User","id":"user:-1","uid":-1,"login":"Former Member","email":"","avatar":null,"rank":null,"kudosWeight":1,"registrationData":{"__typename":"RegistrationData","status":"ANONYMOUS","registrationTime":null,"confirmEmailStatus":false,"registrationAccessLevel":"VIEW","ssoRegistrationFields":[]},"ssoId":null,"profileSettings":{"__typename":"ProfileSettings","dateDisplayStyle":{"__typename":"InheritableStringSettingWithPossibleValues","key":"layout.friendly_dates_enabled","value":"false","localValue":"true","possibleValues":["true","false"]},"dateDisplayFormat":{"__typename":"InheritableStringSetting","key":"layout.format_pattern_date","value":"dd-MMM-yyyy","localValue":"MM-dd-yyyy"},"language":{"__typename":"InheritableStringSettingWithPossibleValues","key":"profile.language","value":"en-US","localValue":null,"possibleValues":["en-US"]}},"deleted":false},"Theme:customTheme1":{"__typename":"Theme","id":"customTheme1"},"CachedAsset:theme:customTheme1-1742462581110":{"__typename":"CachedAsset","id":"theme:customTheme1-1742462581110","value":{"id":"customTheme1","animation":{"fast":"150ms","normal":"250ms","slow":"500ms","slowest":"750ms","function":"cubic-bezier(0.07, 0.91, 0.51, 1)","__typename":"AnimationThemeSettings"},"avatar":{"borderRadius":"50%","collections":["custom"],"__typename":"AvatarThemeSettings"},"basics":{"browserIcon":{"imageAssetName":"JimmyPackets-512-1702592938213.png","imageLastModified":"1702592945815","__typename":"ThemeAsset"},"customerLogo":{"imageAssetName":"f5_logo_fix-1704824537976.svg","imageLastModified":"1704824540697","__typename":"ThemeAsset"},"maximumWidthOfPageContent":"1600px","oneColumnNarrowWidth":"800px","gridGutterWidthMd":"30px","gridGutterWidthXs":"10px","pageWidthStyle":"WIDTH_OF_PAGE_CONTENT","__typename":"BasicsThemeSettings"},"buttons":{"borderRadiusSm":"5px","borderRadius":"5px","borderRadiusLg":"5px","paddingY":"5px","paddingYLg":"7px","paddingYHero":"var(--lia-bs-btn-padding-y-lg)","paddingX":"12px","paddingXLg":"14px","paddingXHero":"42px","fontStyle":"NORMAL","fontWeight":"400","textTransform":"NONE","disabledOpacity":0.5,"primaryTextColor":"var(--lia-bs-white)","primaryTextHoverColor":"var(--lia-bs-white)","primaryTextActiveColor":"var(--lia-bs-white)","primaryBgColor":"var(--lia-bs-primary)","primaryBgHoverColor":"hsl(var(--lia-bs-primary-h), var(--lia-bs-primary-s), calc(var(--lia-bs-primary-l) * 0.85))","primaryBgActiveColor":"hsl(var(--lia-bs-primary-h), var(--lia-bs-primary-s), calc(var(--lia-bs-primary-l) * 0.7))","primaryBorder":"1px solid transparent","primaryBorderHover":"1px solid transparent","primaryBorderActive":"1px solid transparent","primaryBorderFocus":"1px solid var(--lia-bs-white)","primaryBoxShadowFocus":"0 0 0 1px var(--lia-bs-primary), 0 0 0 4px hsla(var(--lia-bs-primary-h), var(--lia-bs-primary-s), var(--lia-bs-primary-l), 0.2)","secondaryTextColor":"var(--lia-bs-gray-900)","secondaryTextHoverColor":"hsl(var(--lia-bs-gray-900-h), var(--lia-bs-gray-900-s), calc(var(--lia-bs-gray-900-l) * 0.95))","secondaryTextActiveColor":"hsl(var(--lia-bs-gray-900-h), var(--lia-bs-gray-900-s), calc(var(--lia-bs-gray-900-l) * 0.9))","secondaryBgColor":"var(--lia-bs-gray-400)","secondaryBgHoverColor":"hsl(var(--lia-bs-gray-400-h), var(--lia-bs-gray-400-s), calc(var(--lia-bs-gray-400-l) * 0.96))","secondaryBgActiveColor":"hsl(var(--lia-bs-gray-400-h), var(--lia-bs-gray-400-s), calc(var(--lia-bs-gray-400-l) * 0.92))","secondaryBorder":"1px solid transparent","secondaryBorderHover":"1px solid transparent","secondaryBorderActive":"1px solid transparent","secondaryBorderFocus":"1px solid transparent","secondaryBoxShadowFocus":"0 0 0 1px var(--lia-bs-primary), 0 0 0 4px hsla(var(--lia-bs-primary-h), var(--lia-bs-primary-s), var(--lia-bs-primary-l), 0.2)","tertiaryTextColor":"var(--lia-bs-gray-900)","tertiaryTextHoverColor":"hsl(var(--lia-bs-gray-900-h), var(--lia-bs-gray-900-s), calc(var(--lia-bs-gray-900-l) * 0.95))","tertiaryTextActiveColor":"hsl(var(--lia-bs-gray-900-h), var(--lia-bs-gray-900-s), calc(var(--lia-bs-gray-900-l) * 0.9))","tertiaryBgColor":"transparent","tertiaryBgHoverColor":"transparent","tertiaryBgActiveColor":"hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.04)","tertiaryBorder":"1px solid transparent","tertiaryBorderHover":"1px solid hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.08)","tertiaryBorderActive":"1px solid transparent","tertiaryBorderFocus":"1px solid transparent","tertiaryBoxShadowFocus":"0 0 0 1px var(--lia-bs-primary), 0 0 0 4px hsla(var(--lia-bs-primary-h), var(--lia-bs-primary-s), var(--lia-bs-primary-l), 0.2)","destructiveTextColor":"var(--lia-bs-danger)","destructiveTextHoverColor":"hsl(var(--lia-bs-danger-h), var(--lia-bs-danger-s), calc(var(--lia-bs-danger-l) * 0.95))","destructiveTextActiveColor":"hsl(var(--lia-bs-danger-h), var(--lia-bs-danger-s), calc(var(--lia-bs-danger-l) * 0.9))","destructiveBgColor":"var(--lia-bs-gray-300)","destructiveBgHoverColor":"hsl(var(--lia-bs-gray-300-h), var(--lia-bs-gray-300-s), calc(var(--lia-bs-gray-300-l) * 0.96))","destructiveBgActiveColor":"hsl(var(--lia-bs-gray-300-h), var(--lia-bs-gray-300-s), calc(var(--lia-bs-gray-300-l) * 0.92))","destructiveBorder":"1px solid transparent","destructiveBorderHover":"1px solid transparent","destructiveBorderActive":"1px solid transparent","destructiveBorderFocus":"1px solid transparent","destructiveBoxShadowFocus":"0 0 0 1px var(--lia-bs-primary), 0 0 0 4px hsla(var(--lia-bs-primary-h), var(--lia-bs-primary-s), var(--lia-bs-primary-l), 0.2)","__typename":"ButtonsThemeSettings"},"border":{"color":"hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.08)","mainContent":"NONE","sideContent":"NONE","radiusSm":"3px","radius":"5px","radiusLg":"9px","radius50":"100vw","__typename":"BorderThemeSettings"},"boxShadow":{"xs":"0 0 0 1px hsla(var(--lia-bs-gray-900-h), var(--lia-bs-gray-900-s), var(--lia-bs-gray-900-l), 0.08), 0 3px 0 -1px hsla(var(--lia-bs-gray-900-h), var(--lia-bs-gray-900-s), var(--lia-bs-gray-900-l), 0.08)","sm":"0 2px 4px hsla(var(--lia-bs-gray-900-h), var(--lia-bs-gray-900-s), var(--lia-bs-gray-900-l), 0.06)","md":"0 5px 15px hsla(var(--lia-bs-gray-900-h), var(--lia-bs-gray-900-s), var(--lia-bs-gray-900-l), 0.15)","lg":"0 10px 30px hsla(var(--lia-bs-gray-900-h), var(--lia-bs-gray-900-s), var(--lia-bs-gray-900-l), 0.15)","__typename":"BoxShadowThemeSettings"},"cards":{"bgColor":"var(--lia-panel-bg-color)","borderRadius":"var(--lia-panel-border-radius)","boxShadow":"var(--lia-box-shadow-xs)","__typename":"CardsThemeSettings"},"chip":{"maxWidth":"300px","height":"30px","__typename":"ChipThemeSettings"},"coreTypes":{"defaultMessageLinkColor":"var(--lia-bs-primary)","defaultMessageLinkDecoration":"none","defaultMessageLinkFontStyle":"NORMAL","defaultMessageLinkFontWeight":"400","defaultMessageFontStyle":"NORMAL","defaultMessageFontWeight":"400","forumColor":"#0C5C8D","forumFontFamily":"var(--lia-bs-font-family-base)","forumFontWeight":"var(--lia-default-message-font-weight)","forumLineHeight":"var(--lia-bs-line-height-base)","forumFontStyle":"var(--lia-default-message-font-style)","forumMessageLinkColor":"var(--lia-default-message-link-color)","forumMessageLinkDecoration":"var(--lia-default-message-link-decoration)","forumMessageLinkFontStyle":"var(--lia-default-message-link-font-style)","forumMessageLinkFontWeight":"var(--lia-default-message-link-font-weight)","forumSolvedColor":"#62C026","blogColor":"#730015","blogFontFamily":"var(--lia-bs-font-family-base)","blogFontWeight":"var(--lia-default-message-font-weight)","blogLineHeight":"1.75","blogFontStyle":"var(--lia-default-message-font-style)","blogMessageLinkColor":"var(--lia-default-message-link-color)","blogMessageLinkDecoration":"var(--lia-default-message-link-decoration)","blogMessageLinkFontStyle":"var(--lia-default-message-link-font-style)","blogMessageLinkFontWeight":"var(--lia-default-message-link-font-weight)","tkbColor":"#C20025","tkbFontFamily":"var(--lia-bs-font-family-base)","tkbFontWeight":"var(--lia-default-message-font-weight)","tkbLineHeight":"1.75","tkbFontStyle":"var(--lia-default-message-font-style)","tkbMessageLinkColor":"var(--lia-default-message-link-color)","tkbMessageLinkDecoration":"var(--lia-default-message-link-decoration)","tkbMessageLinkFontStyle":"var(--lia-default-message-link-font-style)","tkbMessageLinkFontWeight":"var(--lia-default-message-link-font-weight)","qandaColor":"#4099E2","qandaFontFamily":"var(--lia-bs-font-family-base)","qandaFontWeight":"var(--lia-default-message-font-weight)","qandaLineHeight":"var(--lia-bs-line-height-base)","qandaFontStyle":"var(--lia-default-message-link-font-style)","qandaMessageLinkColor":"var(--lia-default-message-link-color)","qandaMessageLinkDecoration":"var(--lia-default-message-link-decoration)","qandaMessageLinkFontStyle":"var(--lia-default-message-link-font-style)","qandaMessageLinkFontWeight":"var(--lia-default-message-link-font-weight)","qandaSolvedColor":"#3FA023","ideaColor":"#F3704B","ideaFontFamily":"var(--lia-bs-font-family-base)","ideaFontWeight":"var(--lia-default-message-font-weight)","ideaLineHeight":"var(--lia-bs-line-height-base)","ideaFontStyle":"var(--lia-default-message-font-style)","ideaMessageLinkColor":"var(--lia-default-message-link-color)","ideaMessageLinkDecoration":"var(--lia-default-message-link-decoration)","ideaMessageLinkFontStyle":"var(--lia-default-message-link-font-style)","ideaMessageLinkFontWeight":"var(--lia-default-message-link-font-weight)","contestColor":"#FCC845","contestFontFamily":"var(--lia-bs-font-family-base)","contestFontWeight":"var(--lia-default-message-font-weight)","contestLineHeight":"var(--lia-bs-line-height-base)","contestFontStyle":"var(--lia-default-message-link-font-style)","contestMessageLinkColor":"var(--lia-default-message-link-color)","contestMessageLinkDecoration":"var(--lia-default-message-link-decoration)","contestMessageLinkFontStyle":"ITALIC","contestMessageLinkFontWeight":"var(--lia-default-message-link-font-weight)","occasionColor":"#EE4B5B","occasionFontFamily":"var(--lia-bs-font-family-base)","occasionFontWeight":"var(--lia-default-message-font-weight)","occasionLineHeight":"var(--lia-bs-line-height-base)","occasionFontStyle":"var(--lia-default-message-font-style)","occasionMessageLinkColor":"var(--lia-default-message-link-color)","occasionMessageLinkDecoration":"var(--lia-default-message-link-decoration)","occasionMessageLinkFontStyle":"var(--lia-default-message-link-font-style)","occasionMessageLinkFontWeight":"var(--lia-default-message-link-font-weight)","grouphubColor":"#491B62","categoryColor":"#949494","communityColor":"#FFFFFF","productColor":"#949494","__typename":"CoreTypesThemeSettings"},"colors":{"black":"#000000","white":"#FFFFFF","gray100":"#F7F7F7","gray200":"#F7F7F7","gray300":"#E8E8E8","gray400":"#D9D9D9","gray500":"#CCCCCC","gray600":"#949494","gray700":"#707070","gray800":"#545454","gray900":"#333333","dark":"#545454","light":"#F7F7F7","primary":"#0C5C8D","secondary":"#333333","bodyText":"#222222","bodyBg":"#F5F5F5","info":"#1D9CD3","success":"#62C026","warning":"#FFD651","danger":"#C20025","alertSystem":"#FF6600","textMuted":"#707070","highlight":"#FFFCAD","outline":"var(--lia-bs-primary)","custom":["#C20025","#081B85","#009639","#B3C6D7","#7CC0EB","#F29A36"],"__typename":"ColorsThemeSettings"},"divider":{"size":"3px","marginLeft":"4px","marginRight":"4px","borderRadius":"50%","bgColor":"var(--lia-bs-gray-600)","bgColorActive":"var(--lia-bs-gray-600)","__typename":"DividerThemeSettings"},"dropdown":{"fontSize":"var(--lia-bs-font-size-sm)","borderColor":"var(--lia-bs-border-color)","borderRadius":"var(--lia-bs-border-radius-sm)","dividerBg":"var(--lia-bs-gray-300)","itemPaddingY":"5px","itemPaddingX":"20px","headerColor":"var(--lia-bs-gray-700)","__typename":"DropdownThemeSettings"},"email":{"link":{"color":"#0069D4","hoverColor":"#0061c2","decoration":"none","hoverDecoration":"underline","__typename":"EmailLinkSettings"},"border":{"color":"#e4e4e4","__typename":"EmailBorderSettings"},"buttons":{"borderRadiusLg":"5px","paddingXLg":"16px","paddingYLg":"7px","fontWeight":"700","primaryTextColor":"#ffffff","primaryTextHoverColor":"#ffffff","primaryBgColor":"#0069D4","primaryBgHoverColor":"#005cb8","primaryBorder":"1px solid transparent","primaryBorderHover":"1px solid transparent","__typename":"EmailButtonsSettings"},"panel":{"borderRadius":"5px","borderColor":"#e4e4e4","__typename":"EmailPanelSettings"},"__typename":"EmailThemeSettings"},"emoji":{"skinToneDefault":"#ffcd43","skinToneLight":"#fae3c5","skinToneMediumLight":"#e2cfa5","skinToneMedium":"#daa478","skinToneMediumDark":"#a78058","skinToneDark":"#5e4d43","__typename":"EmojiThemeSettings"},"heading":{"color":"var(--lia-bs-body-color)","fontFamily":"Inter","fontStyle":"NORMAL","fontWeight":"600","h1FontSize":"30px","h2FontSize":"25px","h3FontSize":"20px","h4FontSize":"18px","h5FontSize":"16px","h6FontSize":"16px","lineHeight":"1.2","subHeaderFontSize":"11px","subHeaderFontWeight":"500","h1LetterSpacing":"normal","h2LetterSpacing":"normal","h3LetterSpacing":"normal","h4LetterSpacing":"normal","h5LetterSpacing":"normal","h6LetterSpacing":"normal","subHeaderLetterSpacing":"2px","h1FontWeight":"var(--lia-bs-headings-font-weight)","h2FontWeight":"var(--lia-bs-headings-font-weight)","h3FontWeight":"var(--lia-bs-headings-font-weight)","h4FontWeight":"var(--lia-bs-headings-font-weight)","h5FontWeight":"var(--lia-bs-headings-font-weight)","h6FontWeight":"var(--lia-bs-headings-font-weight)","__typename":"HeadingThemeSettings"},"icons":{"size10":"10px","size12":"12px","size14":"14px","size16":"16px","size20":"20px","size24":"24px","size30":"30px","size40":"40px","size50":"50px","size60":"60px","size80":"80px","size120":"120px","size160":"160px","__typename":"IconsThemeSettings"},"imagePreview":{"bgColor":"var(--lia-bs-gray-900)","titleColor":"var(--lia-bs-white)","controlColor":"var(--lia-bs-white)","controlBgColor":"var(--lia-bs-gray-800)","__typename":"ImagePreviewThemeSettings"},"input":{"borderColor":"var(--lia-bs-gray-600)","disabledColor":"var(--lia-bs-gray-600)","focusBorderColor":"var(--lia-bs-primary)","labelMarginBottom":"10px","btnFontSize":"var(--lia-bs-font-size-sm)","focusBoxShadow":"0 0 0 3px hsla(var(--lia-bs-primary-h), var(--lia-bs-primary-s), var(--lia-bs-primary-l), 0.2)","checkLabelMarginBottom":"2px","checkboxBorderRadius":"3px","borderRadiusSm":"var(--lia-bs-border-radius-sm)","borderRadius":"var(--lia-bs-border-radius)","borderRadiusLg":"var(--lia-bs-border-radius-lg)","formTextMarginTop":"4px","textAreaBorderRadius":"var(--lia-bs-border-radius)","activeFillColor":"var(--lia-bs-primary)","__typename":"InputThemeSettings"},"loading":{"dotDarkColor":"hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.2)","dotLightColor":"hsla(var(--lia-bs-white-h), var(--lia-bs-white-s), var(--lia-bs-white-l), 0.5)","barDarkColor":"hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.06)","barLightColor":"hsla(var(--lia-bs-white-h), var(--lia-bs-white-s), var(--lia-bs-white-l), 0.4)","__typename":"LoadingThemeSettings"},"link":{"color":"var(--lia-bs-primary)","hoverColor":"hsl(var(--lia-bs-primary-h), var(--lia-bs-primary-s), calc(var(--lia-bs-primary-l) - 10%))","decoration":"none","hoverDecoration":"underline","__typename":"LinkThemeSettings"},"listGroup":{"itemPaddingY":"15px","itemPaddingX":"15px","borderColor":"var(--lia-bs-gray-300)","__typename":"ListGroupThemeSettings"},"modal":{"contentTextColor":"var(--lia-bs-body-color)","contentBg":"var(--lia-bs-white)","backgroundBg":"var(--lia-bs-black)","smSize":"440px","mdSize":"760px","lgSize":"1080px","backdropOpacity":0.3,"contentBoxShadowXs":"var(--lia-bs-box-shadow-sm)","contentBoxShadow":"var(--lia-bs-box-shadow)","headerFontWeight":"700","__typename":"ModalThemeSettings"},"navbar":{"position":"FIXED","background":{"attachment":null,"clip":null,"color":"var(--lia-bs-white)","imageAssetName":null,"imageLastModified":"0","origin":null,"position":"CENTER_CENTER","repeat":"NO_REPEAT","size":"COVER","__typename":"BackgroundProps"},"backgroundOpacity":0.8,"paddingTop":"15px","paddingBottom":"15px","borderBottom":"1px solid var(--lia-bs-border-color)","boxShadow":"var(--lia-bs-box-shadow-sm)","brandMarginRight":"30px","brandMarginRightSm":"10px","brandLogoHeight":"30px","linkGap":"10px","linkJustifyContent":"flex-start","linkPaddingY":"5px","linkPaddingX":"10px","linkDropdownPaddingY":"9px","linkDropdownPaddingX":"var(--lia-nav-link-px)","linkColor":"var(--lia-bs-body-color)","linkHoverColor":"var(--lia-bs-primary)","linkFontSize":"var(--lia-bs-font-size-sm)","linkFontStyle":"NORMAL","linkFontWeight":"400","linkTextTransform":"NONE","linkLetterSpacing":"normal","linkBorderRadius":"var(--lia-bs-border-radius-sm)","linkBgColor":"transparent","linkBgHoverColor":"transparent","linkBorder":"none","linkBorderHover":"none","linkBoxShadow":"none","linkBoxShadowHover":"none","linkTextBorderBottom":"none","linkTextBorderBottomHover":"none","dropdownPaddingTop":"10px","dropdownPaddingBottom":"15px","dropdownPaddingX":"10px","dropdownMenuOffset":"2px","dropdownDividerMarginTop":"10px","dropdownDividerMarginBottom":"10px","dropdownBorderColor":"hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.08)","controllerBgHoverColor":"hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.1)","controllerIconColor":"var(--lia-bs-body-color)","controllerIconHoverColor":"var(--lia-bs-body-color)","controllerTextColor":"var(--lia-nav-controller-icon-color)","controllerTextHoverColor":"var(--lia-nav-controller-icon-hover-color)","controllerHighlightColor":"hsla(30, 100%, 50%)","controllerHighlightTextColor":"var(--lia-yiq-light)","controllerBorderRadius":"var(--lia-border-radius-50)","hamburgerColor":"var(--lia-nav-controller-icon-color)","hamburgerHoverColor":"var(--lia-nav-controller-icon-color)","hamburgerBgColor":"transparent","hamburgerBgHoverColor":"transparent","hamburgerBorder":"none","hamburgerBorderHover":"none","collapseMenuMarginLeft":"20px","collapseMenuDividerBg":"var(--lia-nav-link-color)","collapseMenuDividerOpacity":0.16,"__typename":"NavbarThemeSettings"},"pager":{"textColor":"var(--lia-bs-link-color)","textFontWeight":"var(--lia-font-weight-md)","textFontSize":"var(--lia-bs-font-size-sm)","__typename":"PagerThemeSettings"},"panel":{"bgColor":"var(--lia-bs-white)","borderRadius":"var(--lia-bs-border-radius)","borderColor":"var(--lia-bs-border-color)","boxShadow":"none","__typename":"PanelThemeSettings"},"popover":{"arrowHeight":"8px","arrowWidth":"16px","maxWidth":"300px","minWidth":"100px","headerBg":"var(--lia-bs-white)","borderColor":"var(--lia-bs-border-color)","borderRadius":"var(--lia-bs-border-radius)","boxShadow":"0 0.5rem 1rem hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.15)","__typename":"PopoverThemeSettings"},"prism":{"color":"#000000","bgColor":"#f5f2f0","fontFamily":"var(--font-family-monospace)","fontSize":"var(--lia-bs-font-size-base)","fontWeightBold":"var(--lia-bs-font-weight-bold)","fontStyleItalic":"italic","tabSize":2,"highlightColor":"#b3d4fc","commentColor":"#62707e","punctuationColor":"#6f6f6f","namespaceOpacity":"0.7","propColor":"#990055","selectorColor":"#517a00","operatorColor":"#906736","operatorBgColor":"hsla(0, 0%, 100%, 0.5)","keywordColor":"#0076a9","functionColor":"#d3284b","variableColor":"#c14700","__typename":"PrismThemeSettings"},"rte":{"bgColor":"var(--lia-bs-white)","borderRadius":"var(--lia-panel-border-radius)","boxShadow":" var(--lia-panel-box-shadow)","customColor1":"#bfedd2","customColor2":"#fbeeb8","customColor3":"#f8cac6","customColor4":"#eccafa","customColor5":"#c2e0f4","customColor6":"#2dc26b","customColor7":"#f1c40f","customColor8":"#e03e2d","customColor9":"#b96ad9","customColor10":"#3598db","customColor11":"#169179","customColor12":"#e67e23","customColor13":"#ba372a","customColor14":"#843fa1","customColor15":"#236fa1","customColor16":"#ecf0f1","customColor17":"#ced4d9","customColor18":"#95a5a6","customColor19":"#7e8c8d","customColor20":"#34495e","customColor21":"#000000","customColor22":"#ffffff","defaultMessageHeaderMarginTop":"14px","defaultMessageHeaderMarginBottom":"10px","defaultMessageItemMarginTop":"0","defaultMessageItemMarginBottom":"10px","diffAddedColor":"hsla(170, 53%, 51%, 0.4)","diffChangedColor":"hsla(43, 97%, 63%, 0.4)","diffNoneColor":"hsla(0, 0%, 80%, 0.4)","diffRemovedColor":"hsla(9, 74%, 47%, 0.4)","specialMessageHeaderMarginTop":"14px","specialMessageHeaderMarginBottom":"10px","specialMessageItemMarginTop":"0","specialMessageItemMarginBottom":"10px","__typename":"RteThemeSettings"},"tags":{"bgColor":"var(--lia-bs-gray-200)","bgHoverColor":"var(--lia-bs-gray-400)","borderRadius":"var(--lia-bs-border-radius-sm)","color":"var(--lia-bs-body-color)","hoverColor":"var(--lia-bs-body-color)","fontWeight":"var(--lia-font-weight-md)","fontSize":"var(--lia-font-size-xxs)","textTransform":"UPPERCASE","letterSpacing":"0.5px","__typename":"TagsThemeSettings"},"toasts":{"borderRadius":"var(--lia-bs-border-radius)","paddingX":"12px","__typename":"ToastsThemeSettings"},"typography":{"fontFamilyBase":"Atkinson Hyperlegible","fontStyleBase":"NORMAL","fontWeightBase":"400","fontWeightLight":"300","fontWeightNormal":"400","fontWeightMd":"500","fontWeightBold":"700","letterSpacingSm":"normal","letterSpacingXs":"normal","lineHeightBase":"1.3","fontSizeBase":"15px","fontSizeXxs":"11px","fontSizeXs":"12px","fontSizeSm":"13px","fontSizeLg":"20px","fontSizeXl":"24px","smallFontSize":"14px","customFonts":[],"__typename":"TypographyThemeSettings"},"unstyledListItem":{"marginBottomSm":"5px","marginBottomMd":"10px","marginBottomLg":"15px","marginBottomXl":"20px","marginBottomXxl":"25px","__typename":"UnstyledListItemThemeSettings"},"yiq":{"light":"#ffffff","dark":"#000000","__typename":"YiqThemeSettings"},"colorLightness":{"primaryDark":0.36,"primaryLight":0.74,"primaryLighter":0.89,"primaryLightest":0.95,"infoDark":0.39,"infoLight":0.72,"infoLighter":0.85,"infoLightest":0.93,"successDark":0.24,"successLight":0.62,"successLighter":0.8,"successLightest":0.91,"warningDark":0.39,"warningLight":0.68,"warningLighter":0.84,"warningLightest":0.93,"dangerDark":0.41,"dangerLight":0.72,"dangerLighter":0.89,"dangerLightest":0.95,"__typename":"ColorLightnessThemeSettings"},"localOverride":false,"__typename":"Theme"},"localOverride":false},"CachedAsset:text:en_US-shared/client/components/common/Loading/LoadingDot-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-shared/client/components/common/Loading/LoadingDot-1743097588266","value":{"title":"Loading..."},"localOverride":false},"CachedAsset:text:en_US-components/common/EmailVerification-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/common/EmailVerification-1743097588266","value":{"email.verification.title":"Email Verification Required","email.verification.message.update.email":"To participate in the community, you must first verify your email address. The verification email was sent to {email}. To change your email, visit My Settings.","email.verification.message.resend.email":"To participate in the community, you must first verify your email address. The verification email was sent to {email}. Resend email."},"localOverride":false},"CachedAsset:text:en_US-pages/tags/TagPage-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-pages/tags/TagPage-1743097588266","value":{"tagPageTitle":"Tag:\"{tagName}\" | {communityTitle}","tagPageForNodeTitle":"Tag:\"{tagName}\" in \"{title}\" | {communityTitle}","name":"Tags Page","tag":"Tag: {tagName}"},"localOverride":false},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bi0zNC0xM2k0MzE3N0Q2NjFBRDg5NDAy\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bi0zNC0xM2k0MzE3N0Q2NjFBRDg5NDAy","mimeType":"image/png"},"Category:category:Articles":{"__typename":"Category","id":"category:Articles","entityType":"CATEGORY","displayId":"Articles","nodeType":"category","depth":1,"title":"Articles","shortTitle":"Articles","parent":{"__ref":"Category:category:top"},"categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:top":{"__typename":"Category","id":"category:top","displayId":"top","nodeType":"category","depth":0,"title":"Top"},"Tkb:board:TechnicalArticles":{"__typename":"Tkb","id":"board:TechnicalArticles","entityType":"TKB","displayId":"TechnicalArticles","nodeType":"board","depth":2,"conversationStyle":"TKB","title":"Technical Articles","description":"F5 SMEs share good practice.","avatar":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bi0zNC0xM2k0MzE3N0Q2NjFBRDg5NDAy\"}"},"profileSettings":{"__typename":"ProfileSettings","language":null},"parent":{"__ref":"Category:category:Articles"},"ancestors":{"__typename":"CoreNodeConnection","edges":[{"__typename":"CoreNodeEdge","node":{"__ref":"Community:community:zihoc95639"}},{"__typename":"CoreNodeEdge","node":{"__ref":"Category:category:Articles"}}]},"userContext":{"__typename":"NodeUserContext","canAddAttachments":false,"canUpdateNode":false,"canPostMessages":false,"isSubscribed":false},"boardPolicies":{"__typename":"BoardPolicies","canPublishArticleOnCreate":{"__typename":"PolicyResult","failureReason":{"__typename":"FailureReason","message":"error.lithium.policies.forums.policy_can_publish_on_create_workflow_action.accessDenied","key":"error.lithium.policies.forums.policy_can_publish_on_create_workflow_action.accessDenied","args":[]}},"canReadNode":{"__typename":"PolicyResult","failureReason":null}},"tkbPolicies":{"__typename":"TkbPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}},"shortTitle":"Technical Articles","tagPolicies":{"__typename":"TagPolicies","canSubscribeTagOnNode":{"__typename":"PolicyResult","failureReason":{"__typename":"FailureReason","message":"error.lithium.policies.labels.action.corenode.subscribe_labels.allow.accessDenied","key":"error.lithium.policies.labels.action.corenode.subscribe_labels.allow.accessDenied","args":[]}},"canManageTagDashboard":{"__typename":"PolicyResult","failureReason":{"__typename":"FailureReason","message":"error.lithium.policies.labels.action.corenode.admin_labels.allow.accessDenied","key":"error.lithium.policies.labels.action.corenode.admin_labels.allow.accessDenied","args":[]}}}},"CachedAsset:quilt:f5.prod:pages/tags/TagPage:board:TechnicalArticles-1743097590074":{"__typename":"CachedAsset","id":"quilt:f5.prod:pages/tags/TagPage:board:TechnicalArticles-1743097590074","value":{"id":"TagPage","container":{"id":"Common","headerProps":{"removeComponents":["community.widget.bannerWidget"],"__typename":"QuiltContainerSectionProps"},"items":[{"id":"tag-header-widget","layout":"ONE_COLUMN","bgColor":"var(--lia-bs-white)","showBorder":"BOTTOM","sectionEditLevel":"LOCKED","columnMap":{"main":[{"id":"tags.widget.TagsHeaderWidget","__typename":"QuiltComponent"}],"__typename":"OneSectionColumns"},"__typename":"OneColumnQuiltSection"},{"id":"messages-list-for-tag-widget","layout":"ONE_COLUMN","columnMap":{"main":[{"id":"messages.widget.messageListForNodeByRecentActivityWidget","props":{"viewVariant":{"type":"inline","props":{"useUnreadCount":true,"useViewCount":true,"useAuthorLogin":true,"clampBodyLines":3,"useAvatar":true,"useBoardIcon":false,"useKudosCount":true,"usePreviewMedia":true,"useTags":false,"useNode":true,"useNodeLink":true,"useTextBody":true,"truncateBodyLength":-1,"useBody":true,"useRepliesCount":true,"useSolvedBadge":true,"timeStampType":"conversation.lastPostingActivityTime","useMessageTimeLink":true,"clampSubjectLines":2}},"panelType":"divider","useTitle":false,"hideIfEmpty":false,"pagerVariant":{"type":"loadMore"},"style":"list","showTabs":true,"tabItemMap":{"default":{"mostRecent":true,"mostRecentUserContent":false,"newest":false},"additional":{"mostKudoed":true,"mostViewed":true,"mostReplies":false,"noReplies":false,"noSolutions":false,"solutions":false}}},"__typename":"QuiltComponent"}],"__typename":"OneSectionColumns"},"__typename":"OneColumnQuiltSection"}],"__typename":"QuiltContainer"},"__typename":"Quilt"},"localOverride":false},"CachedAsset:quiltWrapper:f5.prod:Common:1742462479161":{"__typename":"CachedAsset","id":"quiltWrapper:f5.prod:Common:1742462479161","value":{"id":"Common","header":{"backgroundImageProps":{"assetName":"header.jpg","backgroundSize":"COVER","backgroundRepeat":"NO_REPEAT","backgroundPosition":"LEFT_CENTER","lastModified":"1702932449000","__typename":"BackgroundImageProps"},"backgroundColor":"transparent","items":[{"id":"custom.widget.Beta_MetaNav","props":{"widgetVisibility":"signedInOrAnonymous","useTitle":true,"useBackground":false,"title":"","lazyLoad":false},"__typename":"QuiltComponent"},{"id":"community.widget.navbarWidget","props":{"showUserName":false,"showRegisterLink":true,"style":{"boxShadow":"var(--lia-bs-box-shadow-sm)","linkFontWeight":"700","controllerHighlightColor":"hsla(30, 100%, 50%)","dropdownDividerMarginBottom":"10px","hamburgerBorderHover":"none","linkFontSize":"15px","linkBoxShadowHover":"none","backgroundOpacity":0.4,"controllerBorderRadius":"var(--lia-border-radius-50)","hamburgerBgColor":"transparent","linkTextBorderBottom":"none","hamburgerColor":"var(--lia-nav-controller-icon-color)","brandLogoHeight":"48px","linkLetterSpacing":"normal","linkBgHoverColor":"transparent","collapseMenuDividerOpacity":0.16,"paddingBottom":"10px","dropdownPaddingBottom":"15px","dropdownMenuOffset":"2px","hamburgerBgHoverColor":"transparent","borderBottom":"0","hamburgerBorder":"none","dropdownPaddingX":"10px","brandMarginRightSm":"10px","linkBoxShadow":"none","linkJustifyContent":"center","linkColor":"var(--lia-bs-primary)","collapseMenuDividerBg":"var(--lia-nav-link-color)","dropdownPaddingTop":"10px","controllerHighlightTextColor":"var(--lia-yiq-dark)","background":{"imageAssetName":"","color":"var(--lia-bs-white)","size":"COVER","repeat":"NO_REPEAT","position":"CENTER_CENTER","imageLastModified":""},"linkBorderRadius":"var(--lia-bs-border-radius-sm)","linkHoverColor":"var(--lia-bs-primary)","position":"FIXED","linkBorder":"none","linkTextBorderBottomHover":"2px solid #0C5C8D","brandMarginRight":"30px","hamburgerHoverColor":"var(--lia-nav-controller-icon-color)","linkBorderHover":"none","collapseMenuMarginLeft":"20px","linkFontStyle":"NORMAL","linkPaddingX":"10px","paddingTop":"10px","linkPaddingY":"5px","linkTextTransform":"NONE","dropdownBorderColor":"hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.08)","controllerBgHoverColor":"hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.1)","linkDropdownPaddingX":"var(--lia-nav-link-px)","linkBgColor":"transparent","linkDropdownPaddingY":"9px","controllerIconColor":"#0C5C8D","dropdownDividerMarginTop":"10px","linkGap":"10px","controllerIconHoverColor":"#0C5C8D"},"links":{"sideLinks":[],"mainLinks":[{"children":[{"linkType":"INTERNAL","id":"migrated-link-1","params":{"boardId":"TechnicalForum","categoryId":"Forums"},"routeName":"ForumBoardPage"},{"linkType":"INTERNAL","id":"migrated-link-2","params":{"boardId":"WaterCooler","categoryId":"Forums"},"routeName":"ForumBoardPage"}],"linkType":"INTERNAL","id":"migrated-link-0","params":{"categoryId":"Forums"},"routeName":"CategoryPage"},{"children":[{"linkType":"INTERNAL","id":"migrated-link-4","params":{"boardId":"codeshare","categoryId":"CrowdSRC"},"routeName":"TkbBoardPage"},{"linkType":"INTERNAL","id":"migrated-link-5","params":{"boardId":"communityarticles","categoryId":"CrowdSRC"},"routeName":"TkbBoardPage"}],"linkType":"INTERNAL","id":"migrated-link-3","params":{"categoryId":"CrowdSRC"},"routeName":"CategoryPage"},{"children":[{"linkType":"INTERNAL","id":"migrated-link-7","params":{"boardId":"TechnicalArticles","categoryId":"Articles"},"routeName":"TkbBoardPage"},{"linkType":"INTERNAL","id":"article-series","params":{"boardId":"article-series","categoryId":"Articles"},"routeName":"TkbBoardPage"},{"linkType":"INTERNAL","id":"security-insights","params":{"boardId":"security-insights","categoryId":"Articles"},"routeName":"TkbBoardPage"},{"linkType":"INTERNAL","id":"migrated-link-8","params":{"boardId":"DevCentralNews","categoryId":"Articles"},"routeName":"TkbBoardPage"}],"linkType":"INTERNAL","id":"migrated-link-6","params":{"categoryId":"Articles"},"routeName":"CategoryPage"},{"children":[{"linkType":"INTERNAL","id":"migrated-link-10","params":{"categoryId":"CommunityGroups"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"migrated-link-11","params":{"categoryId":"F5-Groups"},"routeName":"CategoryPage"}],"linkType":"INTERNAL","id":"migrated-link-9","params":{"categoryId":"GroupsCategory"},"routeName":"CategoryPage"},{"children":[],"linkType":"INTERNAL","id":"migrated-link-12","params":{"boardId":"Events","categoryId":"top"},"routeName":"EventBoardPage"},{"children":[],"linkType":"INTERNAL","id":"migrated-link-13","params":{"boardId":"Suggestions","categoryId":"top"},"routeName":"IdeaBoardPage"},{"children":[],"linkType":"EXTERNAL","id":"Common-external-link","url":"https://community.f5.com/c/how-do-i","target":"SELF"}]},"className":"QuiltComponent_lia-component-edit-mode__lQ9Z6","showSearchIcon":false},"__typename":"QuiltComponent"},{"id":"community.widget.bannerWidget","props":{"backgroundColor":"transparent","visualEffects":{"showBottomBorder":false},"backgroundImageProps":{"backgroundSize":"COVER","backgroundPosition":"CENTER_CENTER","backgroundRepeat":"NO_REPEAT"},"fontColor":"#222222"},"__typename":"QuiltComponent"},{"id":"community.widget.breadcrumbWidget","props":{"backgroundColor":"var(--lia-bs-primary)","linkHighlightColor":"#FFFFFF","visualEffects":{"showBottomBorder":false},"backgroundOpacity":60,"linkTextColor":"#FFFFFF"},"__typename":"QuiltComponent"}],"__typename":"QuiltWrapperSection"},"footer":{"backgroundImageProps":{"assetName":null,"backgroundSize":"COVER","backgroundRepeat":"NO_REPEAT","backgroundPosition":"CENTER_CENTER","lastModified":null,"__typename":"BackgroundImageProps"},"backgroundColor":"var(--lia-bs-body-color)","items":[{"id":"custom.widget.Beta_Footer","props":{"widgetVisibility":"signedInOrAnonymous","useTitle":true,"useBackground":false,"title":"","lazyLoad":false},"__typename":"QuiltComponent"},{"id":"custom.widget.Tag_Manager_Helper","props":{"widgetVisibility":"signedInOrAnonymous","useTitle":true,"useBackground":false,"title":"","lazyLoad":false},"__typename":"QuiltComponent"},{"id":"custom.widget.Consent_Blackbar","props":{"widgetVisibility":"signedInOrAnonymous","useTitle":true,"useBackground":false,"title":"","lazyLoad":false},"__typename":"QuiltComponent"}],"__typename":"QuiltWrapperSection"},"__typename":"QuiltWrapper","localOverride":false},"localOverride":false},"CachedAsset:text:en_US-components/common/ActionFeedback-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/common/ActionFeedback-1743097588266","value":{"joinedGroupHub.title":"Welcome","joinedGroupHub.message":"You are now a member of this group and are subscribed to updates.","groupHubInviteNotFound.title":"Invitation Not Found","groupHubInviteNotFound.message":"Sorry, we could not find your invitation to the group. The owner may have canceled the invite.","groupHubNotFound.title":"Group Not Found","groupHubNotFound.message":"The grouphub you tried to join does not exist. It may have been deleted.","existingGroupHubMember.title":"Already Joined","existingGroupHubMember.message":"You are already a member of this group.","accountLocked.title":"Account Locked","accountLocked.message":"Your account has been locked due to multiple failed attempts. Try again in {lockoutTime} minutes.","editedGroupHub.title":"Changes Saved","editedGroupHub.message":"Your group has been updated.","leftGroupHub.title":"Goodbye","leftGroupHub.message":"You are no longer a member of this group and will not receive future updates.","deletedGroupHub.title":"Deleted","deletedGroupHub.message":"The group has been deleted.","groupHubCreated.title":"Group Created","groupHubCreated.message":"{groupHubName} is ready to use","accountClosed.title":"Account Closed","accountClosed.message":"The account has been closed and you will now be redirected to the homepage","resetTokenExpired.title":"Reset Password Link has Expired","resetTokenExpired.message":"Try resetting your password again","invalidUrl.title":"Invalid URL","invalidUrl.message":"The URL you're using is not recognized. Verify your URL and try again.","accountClosedForUser.title":"Account Closed","accountClosedForUser.message":"{userName}'s account is closed","inviteTokenInvalid.title":"Invitation Invalid","inviteTokenInvalid.message":"Your invitation to the community has been canceled or expired.","inviteTokenError.title":"Invitation Verification Failed","inviteTokenError.message":"The url you are utilizing is not recognized. Verify your URL and try again","pageNotFound.title":"Access Denied","pageNotFound.message":"You do not have access to this area of the community or it doesn't exist","eventAttending.title":"Responded as Attending","eventAttending.message":"You'll be notified when there's new activity and reminded as the event approaches","eventInterested.title":"Responded as Interested","eventInterested.message":"You'll be notified when there's new activity and reminded as the event approaches","eventNotFound.title":"Event Not Found","eventNotFound.message":"The event you tried to respond to does not exist.","redirectToRelatedPage.title":"Showing Related Content","redirectToRelatedPageForBaseUsers.title":"Showing Related Content","redirectToRelatedPageForBaseUsers.message":"The content you are trying to access is archived","redirectToRelatedPage.message":"The content you are trying to access is archived","relatedUrl.archivalLink.flyoutMessage":"The content you are trying to access is archived View Archived Content"},"localOverride":false},"CachedAsset:component:custom.widget.Beta_MetaNav-en-1742462597562":{"__typename":"CachedAsset","id":"component:custom.widget.Beta_MetaNav-en-1742462597562","value":{"component":{"id":"custom.widget.Beta_MetaNav","template":{"id":"Beta_MetaNav","markupLanguage":"HANDLEBARS","style":null,"texts":null,"defaults":{"config":{"applicablePages":[],"description":"MetaNav menu at the top of every page.","fetchedContent":null,"__typename":"ComponentConfiguration"},"props":[],"__typename":"ComponentProperties"},"components":[{"id":"custom.widget.Beta_MetaNav","form":null,"config":null,"props":[],"__typename":"Component"}],"grouping":"CUSTOM","__typename":"ComponentTemplate"},"properties":{"config":{"applicablePages":[],"description":"MetaNav menu at the top of every page.","fetchedContent":null,"__typename":"ComponentConfiguration"},"props":[],"__typename":"ComponentProperties"},"form":null,"__typename":"Component","localOverride":false},"globalCss":null,"form":null},"localOverride":false},"CachedAsset:component:custom.widget.Beta_Footer-en-1742462597562":{"__typename":"CachedAsset","id":"component:custom.widget.Beta_Footer-en-1742462597562","value":{"component":{"id":"custom.widget.Beta_Footer","template":{"id":"Beta_Footer","markupLanguage":"HANDLEBARS","style":null,"texts":null,"defaults":{"config":{"applicablePages":[],"description":"DevCentral´s custom footer.","fetchedContent":null,"__typename":"ComponentConfiguration"},"props":[],"__typename":"ComponentProperties"},"components":[{"id":"custom.widget.Beta_Footer","form":null,"config":null,"props":[],"__typename":"Component"}],"grouping":"CUSTOM","__typename":"ComponentTemplate"},"properties":{"config":{"applicablePages":[],"description":"DevCentral´s custom footer.","fetchedContent":null,"__typename":"ComponentConfiguration"},"props":[],"__typename":"ComponentProperties"},"form":null,"__typename":"Component","localOverride":false},"globalCss":null,"form":null},"localOverride":false},"CachedAsset:component:custom.widget.Tag_Manager_Helper-en-1742462597562":{"__typename":"CachedAsset","id":"component:custom.widget.Tag_Manager_Helper-en-1742462597562","value":{"component":{"id":"custom.widget.Tag_Manager_Helper","template":{"id":"Tag_Manager_Helper","markupLanguage":"HANDLEBARS","style":null,"texts":null,"defaults":{"config":{"applicablePages":[],"description":"Helper widget to inject Tag Manager scripts into head element","fetchedContent":null,"__typename":"ComponentConfiguration"},"props":[],"__typename":"ComponentProperties"},"components":[{"id":"custom.widget.Tag_Manager_Helper","form":null,"config":null,"props":[],"__typename":"Component"}],"grouping":"CUSTOM","__typename":"ComponentTemplate"},"properties":{"config":{"applicablePages":[],"description":"Helper widget to inject Tag Manager scripts into head element","fetchedContent":null,"__typename":"ComponentConfiguration"},"props":[],"__typename":"ComponentProperties"},"form":null,"__typename":"Component","localOverride":false},"globalCss":null,"form":null},"localOverride":false},"CachedAsset:component:custom.widget.Consent_Blackbar-en-1742462597562":{"__typename":"CachedAsset","id":"component:custom.widget.Consent_Blackbar-en-1742462597562","value":{"component":{"id":"custom.widget.Consent_Blackbar","template":{"id":"Consent_Blackbar","markupLanguage":"HTML","style":null,"texts":null,"defaults":{"config":{"applicablePages":[],"description":"","fetchedContent":null,"__typename":"ComponentConfiguration"},"props":[],"__typename":"ComponentProperties"},"components":[{"id":"custom.widget.Consent_Blackbar","form":null,"config":null,"props":[],"__typename":"Component"}],"grouping":"TEXTHTML","__typename":"ComponentTemplate"},"properties":{"config":{"applicablePages":[],"description":"","fetchedContent":null,"__typename":"ComponentConfiguration"},"props":[],"__typename":"ComponentProperties"},"form":null,"__typename":"Component","localOverride":false},"globalCss":null,"form":null},"localOverride":false},"CachedAsset:text:en_US-components/community/Breadcrumb-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/community/Breadcrumb-1743097588266","value":{"navLabel":"Breadcrumbs","dropdown":"Additional parent page navigation"},"localOverride":false},"CachedAsset:text:en_US-components/tags/TagsHeaderWidget-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/tags/TagsHeaderWidget-1743097588266","value":{"tag":"{tagName}","topicsCount":"{count} {count, plural, one {Topic} other {Topics}}"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageListForNodeByRecentActivityWidget-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageListForNodeByRecentActivityWidget-1743097588266","value":{"title@userScope:other":"Recent Content","title@userScope:self":"Contributions","title@board:FORUM@userScope:other":"Recent Discussions","title@board:BLOG@userScope:other":"Recent Blogs","emptyDescription":"No content to show","MessageListForNodeByRecentActivityWidgetEditor.nodeScope.label":"Scope","title@instance:1706288370055":"Content Feed","title@instance:1743095186784":"Most Recent Updates","title@instance:1704317906837":"Content Feed","title@instance:1743095018194":"Most Recent Updates","title@instance:1702668293472":"Community Feed","title@instance:1743095117047":"Most Recent Updates","title@instance:1704319314827":"Blog Feed","title@instance:1743095235555":"Most Recent Updates","title@instance:1704320290851":"My Contributions","title@instance:1703720491809":"Forum Feed","title@instance:1743095311723":"Most Recent Updates","title@instance:1703028709746":"Group Content Feed","title@instance:VTsglH":"Content Feed"},"localOverride":false},"Category:category:Forums":{"__typename":"Category","id":"category:Forums","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Forum:board:TechnicalForum":{"__typename":"Forum","id":"board:TechnicalForum","forumPolicies":{"__typename":"ForumPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}},"boardPolicies":{"__typename":"BoardPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Forum:board:WaterCooler":{"__typename":"Forum","id":"board:WaterCooler","forumPolicies":{"__typename":"ForumPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}},"boardPolicies":{"__typename":"BoardPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Tkb:board:DevCentralNews":{"__typename":"Tkb","id":"board:DevCentralNews","tkbPolicies":{"__typename":"TkbPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}},"boardPolicies":{"__typename":"BoardPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:GroupsCategory":{"__typename":"Category","id":"category:GroupsCategory","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:F5-Groups":{"__typename":"Category","id":"category:F5-Groups","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:CommunityGroups":{"__typename":"Category","id":"category:CommunityGroups","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Occasion:board:Events":{"__typename":"Occasion","id":"board:Events","boardPolicies":{"__typename":"BoardPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}},"occasionPolicies":{"__typename":"OccasionPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Idea:board:Suggestions":{"__typename":"Idea","id":"board:Suggestions","boardPolicies":{"__typename":"BoardPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}},"ideaPolicies":{"__typename":"IdeaPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:CrowdSRC":{"__typename":"Category","id":"category:CrowdSRC","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Tkb:board:codeshare":{"__typename":"Tkb","id":"board:codeshare","tkbPolicies":{"__typename":"TkbPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}},"boardPolicies":{"__typename":"BoardPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Tkb:board:communityarticles":{"__typename":"Tkb","id":"board:communityarticles","tkbPolicies":{"__typename":"TkbPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}},"boardPolicies":{"__typename":"BoardPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Tkb:board:security-insights":{"__typename":"Tkb","id":"board:security-insights","tkbPolicies":{"__typename":"TkbPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}},"boardPolicies":{"__typename":"BoardPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Tkb:board:article-series":{"__typename":"Tkb","id":"board:article-series","tkbPolicies":{"__typename":"TkbPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}},"boardPolicies":{"__typename":"BoardPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Conversation:conversation:333352":{"__typename":"Conversation","id":"conversation:333352","topic":{"__typename":"TkbTopicMessage","uid":333352},"lastPostingActivityTime":"2025-03-15T15:25:10.817-07:00","solved":false},"User:user:303102":{"__typename":"User","uid":303102,"login":"Ulises_Alonso","registrationData":{"__typename":"RegistrationData","status":null},"deleted":false,"avatar":{"__typename":"UserAvatar","url":"https://community.f5.com/t5/s/zihoc95639/images/dS0zMDMxMDItNnNGUVZV?image-coordinates=90%2C0%2C517%2C427"},"id":"user:303102"},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItSlpZcFhn?revision=6\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItSlpZcFhn?revision=6","title":"keeping client source ip.png","associationType":"BODY","width":2814,"height":2119,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItQXpyRXly?revision=6\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItQXpyRXly?revision=6","title":"Screenshot 2024-08-21 at 10.50.19.png","associationType":"BODY","width":3840,"height":2160,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTIteFlFRGIz?revision=6\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTIteFlFRGIz?revision=6","title":"Screenshot 2024-08-21 at 10.59.54.png","associationType":"BODY","width":1396,"height":1672,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTIteUFOOWNP?revision=6\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTIteUFOOWNP?revision=6","title":"Screenshot 2024-08-21 at 11.24.47.png","associationType":"BODY","width":3840,"height":2160,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItVGoyRExx?revision=6\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItVGoyRExx?revision=6","title":"Screenshot 2024-08-21 at 11.43.30.png","associationType":"BODY","width":3840,"height":2160,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItd3RzbTdF?revision=6\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItd3RzbTdF?revision=6","title":"Screenshot 2024-08-21 at 11.48.24.png","associationType":"BODY","width":1346,"height":1652,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItUkMxNGFh?revision=6\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItUkMxNGFh?revision=6","title":"Screenshot 2024-08-21 at 12.10.29.png","associationType":"BODY","width":1720,"height":472,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItcHpWbWVT?revision=6\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItcHpWbWVT?revision=6","title":"Screenshot 2024-08-21 at 12.07.07.png","associationType":"BODY","width":1014,"height":1296,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItbFJKWjBa?revision=6\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItbFJKWjBa?revision=6","title":"Screenshot 2024-08-21 at 11.52.33.png","associationType":"BODY","width":1372,"height":594,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItV1ZEb3g5?revision=6\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItV1ZEb3g5?revision=6","title":"Screenshot 2024-08-21 at 12.13.46.png","associationType":"BODY","width":3840,"height":2160,"altText":""},"TkbTopicMessage:message:333352":{"__typename":"TkbTopicMessage","subject":"F5 BIG-IP deployment with Red Hat OpenShift - keeping client IP addresses and egress flows","conversation":{"__ref":"Conversation:conversation:333352"},"id":"message:333352","revisionNum":6,"uid":333352,"depth":0,"board":{"__ref":"Tkb:board:TechnicalArticles"},"author":{"__ref":"User:user:303102"},"teaser@stripHtml({\"removeProcessingText\":true,\"truncateLength\":-1})":" Controlling the egress traffic in OpenShift allows to use the BIG-IP for several use cases: \n \n Keeping the source IP of the ingress clients \n Providing highly scalable SNAT for egress flows \n Providing security functionalities for egress flows \n ","introduction":"","metrics":{"__typename":"MessageMetrics","views":431},"postTime":"2024-09-02T05:00:00.038-07:00","lastPublishTime":"2025-01-28T10:32:23.794-08:00","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})":" Introduction \n In this article it will be covered how to control the egress traffic in the Red Hat OpenShift cluster making use of the AdminPolicyBasedExternalRoute resource type introduced with OpenShift 4.14. This functionality can be used for several use cases: \n \n Keeping the source IP of the ingress clients by setting the BIG-IP as default gateway for the namespaces which the BIG-IP provides load balancing. \n Providing highly scalable SNAT for egress flows to outside the cluster. This includes per namespace SNATs. \n Providing security functionalities for egress flows to outside the cluster, such as firewall, IDS/IPS, malware protection, reporting and analytics, data loss prevention, URL filtering and integration with third party security solutions by doing SSL/TLS inspection. \n \n Keeping the source IP of ingress clients \n By default OpenShift sends the traffic to the default gateway of the network where the OpenShift cluster is placed. In the case of ingress traffic to the applications, in order to make the traffic symmetric, making the return traffic go through the load balancer, it has been required to use SNAT in the load balancer. With the introduction AdminPolicyBasedExternalRoute this is no longer required, allowing the use the source IP to the workload PODs (in the case of 1-tier deployment) or to the in-cluster ingress controller (in the case of 2-tier deployments). The overall traffic flows in the platform would be as shown next: \n \n Using the feature is straight forward, we only have to specify the namespaces for which we want the BIG-IP to be the gateway and the floating IP of the BIG-IPs facing the OpenShift cluster. An example is shown next: \n apiVersion: k8s.ovn.org/v1 kind: AdminPolicyBasedExternalRoute metadata: name: meg-policy-common spec: from: namespaceSelector: matchLabels: meg: common nextHops: static: - ip: \"10.1.10.100\" \n The labels can be freely chosen, in this case it is chosen \"meg\" as label (meg standing multi-egress gateway) and the value \"common.\" In the next sections we will see further possibilities. \n Providing highly scalable SNAT for egress flows \n One egress SNAT for all namespaces handled by the BIG-IP \n Another use case of this functionality is to delegate the SNAT function into BIG-IP, this allows for highly scalability SNAT for egress flows, this is traffic initiated from the cluster to the outside which is not load balanced. The scalability benefits are two fold: \n \n BIG-IP has a carrier-grade NAT which allows high number of concurrent flows and great configuration flexibility. \n By doing the SNAT function in BIG-IP, it is possible to SNAT the traffic of all nodes to a single IP instead of assigning one SNAT address per node. This allows managing the traffic through firewalls easier because filters don´t need to be changed when nodes are added or removed. \n \n Besides of the above, recall that this configuration in the BIG-IP can be done, if needed, in a per namespace basis, as shown next. \n \n This setup requires a virtual server which typically is not modified and doesn´t need to be created with CIS (it could be created with CIS with an AS3 ConfigMap if desired). The configuration is fairly simple: \n \n From this configuration it is worth highlighting: \n \n Specifying the VS type as Forwarding (IP) \n Restricting the source IPs with the range of the PODs \n Specifying the desired SNAT pool \n \n Multiple egress SNATs for namespaces handled by the BIG-IP \n It might be desired to have different sets of SNATs depending on the namespaces, this can be used by external firewalls to differentiate between applications. This can be accomplished by means of having multiple virtual servers as shown next: \n \n \n In OpenShift, there will be a separate AdminPolicyBasedExternalRoute manifest for each set of namespaces where we want to apply separate SNAT pools. In the figure above, these manifests are the \"meg-policy-common\" shown above and a new \"meg-policy-deciated\" shown next: \n apiVersion: k8s.ovn.org/v1 kind: AdminPolicyBasedExternalRoute metadata: name: meg-policy-dedicated spec: from: namespaceSelector: matchLabels: meg: dedicated nextHops: static: - ip: \"10.1.10.200\" Each AdminPolicyBasedExternalRoute points to a different floating-IP in the BIG-IP. But the BIG-IP cannot differentiate in the forwarding VS in which floating-IP the traffic was received because all the IPs in a given VLAN use the same MAC address. Because of the latter, each virtual server needs to be listening in a different VLAN so the BIG-IP can differentiate from which namespace the traffic came from. As you might have noted, typically the OpenShift nodes only sit in a single VLAN. To solve this, dummy VLANs in the BIG-IP will be created connecting to the same OpenShift network but with different interfaces. Furthermore, in order to have the same subnet in these VLANs it will be needed to assign each VLAN to a different route domain. \n Overall, the setup will be as follows: \n \n In practice the configuration is very similar to the case where there is a single egress virtual server: \n \n Note that from the previous configuration we only had to add the route domain ID (in this case %10) to the IP addresses. The routing domains created (10 and 20) are shown next: \n \n The full configuration of one of these route domains is shown next as example. The only required parameter is to have as parent the route domain 0 because it will be used to reach the external VLAN. \n \n The self-IPs of the BIG-IP facing the OpenShift clusters have to be in the corresponding route domains: \n \n Note that it is required to have separate non-floating IP addresses as well. \n Providing security functionalities for egress flows \n Once the BIG-IP is the gateway of the applications, it can be used to add many functionalities. This is outlined in the next figure: \n \n \n These functionalities are delivered with the SSL Ochestrator and Secure Web Gateway Services add-on modules. \n Furthermore, these modules allow the inclusion of additional third-party security integrations, like Cisco Firepower, Cisco WSA, Symantec DLP, RSA Netwitness, FireEye NX and PaloAlto NG Firewall. \n Conclusion and final remarks \n The AdminPolicyBasedExternalRoute resource type introduced with OpenShift 4.14 opens many possibilities of getting more visibility into the cluster, higher scalability and many security functionalities available in your existing BIG-IP platform. I hope this article has been an eye opener for the possibilities of the BIG-IP platform with OpenShift. I would love to hear if you have any specific requirements for the use cases mentioned in the article. ","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})@stringLength":"6892","kudosSumWeight":1,"repliesCount":1,"readOnly":false,"images":{"__typename":"AssociatedImageConnection","edges":[{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDE","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItSlpZcFhn?revision=6\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDI","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItQXpyRXly?revision=6\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDM","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTIteFlFRGIz?revision=6\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDQ","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTIteUFOOWNP?revision=6\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDU","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItVGoyRExx?revision=6\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDY","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItd3RzbTdF?revision=6\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDc","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItUkMxNGFh?revision=6\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDg","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItcHpWbWVT?revision=6\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDk","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItbFJKWjBa?revision=6\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDEw","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzMzNTItV1ZEb3g5?revision=6\"}"}}],"totalCount":10,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}},"videos":{"__typename":"VideoConnection","edges":[],"totalCount":0,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}}},"Conversation:conversation:339884":{"__typename":"Conversation","id":"conversation:339884","topic":{"__typename":"TkbTopicMessage","uid":339884},"lastPostingActivityTime":"2025-02-24T19:51:18.469-08:00","solved":false},"User:user:421692":{"__typename":"User","uid":421692,"login":"Jennifer_Yeung","registrationData":{"__typename":"RegistrationData","status":null},"deleted":false,"avatar":{"__typename":"UserAvatar","url":"https://community.f5.com/t5/s/zihoc95639/images/dS00MjE2OTItMjM4NjlpMkQ2RjRFMUZDNDMxN0IxRA"},"id":"user:421692"},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtWVU1amF6?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtWVU1amF6?revision=25","title":"pic1.png","associationType":"BODY","width":2782,"height":1392,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtYXEzVmNL?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtYXEzVmNL?revision=25","title":"new-pic2.png","associationType":"BODY","width":2932,"height":1472,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtSmJNZ3Ry?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtSmJNZ3Ry?revision=25","title":"pic3.png","associationType":"BODY","width":1343,"height":751,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtSVJiZ3dK?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtSVJiZ3dK?revision=25","title":"pic4.png","associationType":"BODY","width":1098,"height":661,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtTHA2SDc0?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtTHA2SDc0?revision=25","title":"pic5.png","associationType":"BODY","width":1032,"height":265,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtdTFySzhB?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtdTFySzhB?revision=25","title":"pic6.png","associationType":"BODY","width":1289,"height":380,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtN1dLVFhZ?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtN1dLVFhZ?revision=25","title":"pic7.png","associationType":"BODY","width":1315,"height":736,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtS3RzbXJl?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtS3RzbXJl?revision=25","title":"pic8.png","associationType":"BODY","width":1211,"height":744,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtc0l2OTBl?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtc0l2OTBl?revision=25","title":"new-pic10.png","associationType":"BODY","width":2306,"height":1119,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtNGlPcnZQ?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtNGlPcnZQ?revision=25","title":"new-pic11.png","associationType":"BODY","width":1917,"height":1060,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtbUtVeVYz?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtbUtVeVYz?revision=25","title":"new-pic12.png","associationType":"BODY","width":2460,"height":1262,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtRVBsOVg3?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtRVBsOVg3?revision=25","title":"pic13.png","associationType":"BODY","width":2860,"height":1404,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtdUJkT0V0?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtdUJkT0V0?revision=25","title":"pic14.png","associationType":"BODY","width":2534,"height":416,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtd2YxMEQ3?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtd2YxMEQ3?revision=25","title":"pic15.png","associationType":"BODY","width":3178,"height":1176,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtZDF2R25C?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtZDF2R25C?revision=25","title":"pic16-new.png","associationType":"BODY","width":3074,"height":574,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtdGV0Sk53?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtdGV0Sk53?revision=25","title":"pic17-new.png","associationType":"BODY","width":1920,"height":892,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtZjdZOEU3?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtZjdZOEU3?revision=25","title":"pic18.png","associationType":"BODY","width":2762,"height":1580,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtV2xodXRO?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtV2xodXRO?revision=25","title":"pic17.png","associationType":"BODY","width":998,"height":720,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtS2ZzOW5O?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtS2ZzOW5O?revision=25","title":"pic18.png","associationType":"BODY","width":1730,"height":249,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQta2xHUGlO?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQta2xHUGlO?revision=25","title":"pic19.png","associationType":"BODY","width":1018,"height":851,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtbmp0UFR4?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtbmp0UFR4?revision=25","title":"pic22.png","associationType":"BODY","width":2754,"height":1646,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtZU05eG1o?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtZU05eG1o?revision=25","title":"pic21-a.png","associationType":"BODY","width":1568,"height":607,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtWE1saWtw?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtWE1saWtw?revision=25","title":"pic21.png","associationType":"BODY","width":1197,"height":417,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtTllHYk9Z?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtTllHYk9Z?revision=25","title":"pic24.png","associationType":"BODY","width":2968,"height":466,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtclVkT3E3?revision=25\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtclVkT3E3?revision=25","title":"pic25.png","associationType":"BODY","width":1814,"height":366,"altText":""},"TkbTopicMessage:message:339884":{"__typename":"TkbTopicMessage","subject":"Secure AI RAG using F5 Distributed Cloud in Red Hat OpenShift AI and NetApp ONTAP Environment","conversation":{"__ref":"Conversation:conversation:339884"},"id":"message:339884","revisionNum":25,"uid":339884,"depth":0,"board":{"__ref":"Tkb:board:TechnicalArticles"},"author":{"__ref":"User:user:421692"},"teaser@stripHtml({\"removeProcessingText\":true,\"truncateLength\":-1})":"","introduction":"","metrics":{"__typename":"MessageMetrics","views":304},"postTime":"2025-02-24T19:51:18.469-08:00","lastPublishTime":"2025-02-24T19:51:18.469-08:00","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})":" Introduction \n Retrieval Augmented Generation (RAG) is a powerful technique that allows Large Language Models (LLMs) to access information beyond their training data. The “R” in RAG refers to the data retrieval process, where the system retrieves relevant information from an external knowledge base based on the input query. Next, the “A” in RAG represents the augmentation of context enrichment, as the system combines the retrieved relevant information and the input query to create a more comprehensive prompt for the LLM. Lastly, the “G” in RAG stands for response generation, where the LLM generates a response with a more contextually accurate output based on the augmented prompt as a result. \n RAG is becoming increasingly popular in enterprise AI applications due to its ability to provide more accurate and contextually relevant responses to a wide range of queries. However, deploying RAG can introduce complexity due to its components being located in different environments. For instance, the datastore or corpus, which is a collection of data, is typically on-premise for enhanced control over data access and management due to data security, governance, and compliance with regulations within the enterprise. Meanwhile, inference services are often deployed in the cloud for their scalability and cost-effectiveness. \n In this article, we will discuss how F5 Distributed Cloud can simplify the complexity and securely connect all RAG components seamlessly for enterprise RAG-enabled AI applications deployments. Specifically, we will focus on Network Connect, App Connect, and Web App & API Protection. We will demonstrate how these F5 Distributed Cloud features can be leveraged to secure RAG in collaboration with Red Hat OpenShift AI and NetApp ONTAP. \n Example Topology \n \n \n F5 Distributed Cloud Network Connect \n F5 Distributed Cloud Network Connect enables seamless and secure network connectivity across hybrid and multicloud environments. By deploying F5 Distributed Cloud Customer Edge (CE) at site, it allows us to easily establish encrypted site-to-site connectivity across on-premises, multi-cloud, and edge environment. \n Jensen Huang, CEO of NVIDIA, has said that \"Nearly half of the files in the world are stored on-prem on NetApp.”. In our example, enterprise data stores are deployed on NetApp ONTAP in a data center in Seattle managed by organization B (Segment-B: s-gorman-production-segment), while RAG services, including embedding Large Language Model (LLM) and vector database, is deployed on-premise on a Red Hat OpenShift cluster in a data center in California managed by Organization A (Segment-A: jy-ocp). By leveraging F5 Distributed Cloud Network Connect, we can quickly and easily establish a secure connection for seamless and efficient data transfer from the enterprise data stores to RAG services between these two segments only: \n \n F5 Distributed Cloud CE can be deployed as a virtual machine (VM) or as a pod on a Red Hat OpenShift cluster. In California, we deploy the CE as a VM using Red Hat OpenShift Virtualization — click here to find out more on Deploying F5 Distributed Cloud Customer Edge in Red Hat OpenShift Virtualization: \n \n Segment-A: jy-ocp on CE in California and Segment-B: s-gorman-production-segment on CE in Seattle: \n \n Simply and securely connect Segment-A: jy-ocp and Segment-B: s-gorman-production-segment only, using Segment Connector: \n \n NetApp ONTAP in Seattle has a LUN named “tbd-RAG”, which serves as the enterprise data store in our demo setup and contains a collection of data. After these two data centers are connected using F5 XC Network Connect, a secure encrypted end-to-end connection is established between them. In our example, “test-ai-tbd” is in the data center in California where it hosts the RAG services, including embedding Large Language Model (LLM) and vector database, and it can now successfully connect to the enterprise data stores on NetApp ONTAP in the data center in Seattle: \n \n F5 Distributed Cloud App Connect \n F5 Distributed Cloud App Connect securely connects and delivers distributed applications and services across hybrid and multicloud environments. By utilizing F5 Distributed Cloud App Connect, we can direct the inference traffic through F5 Distributed Cloud's security layers to safeguard our inference endpoints. \n \n Red Hat OpenShift on Amazon Web Services (ROSA) is a fully managed service that allows users to develop, run, and scale applications in a native AWS environment. We can host our inference service on ROSA so that we can leverage the scalability, cost-effectiveness, and numerous benefits of AWS’s managed infrastructure services. For instance, we can host our inference service on ROSA by deploying Ollama with multiple AI/ML models: \n \n Or, we can enable Model Serving on Red Hat OpenShift AI (RHOAI). Red Hat OpenShift AI (RHOAI) is a flexible and scalable AI/ML platform builds on the capabilities of Red Hat OpenShift that facilitates collaboration among data scientists, engineers, and app developers. This platform allows them to serve, build, train, deploy, test, and monitor AI/ML models and applications either on-premise or in the cloud, fostering efficient innovation within organizations. In our example, we use Red Hat OpenShift AI (RHOAI) Model Serving on ROSA for our inference service: \n \n Once inference service is deployed on ROSA, we can utilize F5 Distributed Cloud to secure our inference endpoint by steering the inference traffic through F5 Distributed Cloud's security layers, which offers an extensive suite of features designed specifically for the security of modern AI/ML inference endpoints. This setup would allow us to scrutinize requests, implement policies for detected threats, and protect sensitive datasets before they reach the inferencing service hosted within ROSA. In our example, we setup a F5 Distributed Cloud HTTP Load Balancer (rhoai-llm-serving.f5-demo.com), and we advertise it to the CE in the datacenter in California only: \n \n We now reach our Red Hat OpenShift AI (RHOAI) inference endpoint through F5 Distributed Cloud: \n \n F5 Distributed Cloud Web App & API Protection \n F5 Distributed Cloud Web App & API Protection provides comprehensive sets of security features, and uniform observability and policy enforcement to protect apps and APIs across hybrid and multicloud environments. We utilize F5 Distributed Cloud App Connect to steer the inference traffic through F5 Distributed Cloud to secure our inference endpoint. In our example, we protect our Red Hat OpenShift AI (RHOAI) inference endpoint by rate-limiting the access, so that we can ensure no single client would exhaust the inference service: \n \n A \"Too Many Requests\" is received in the response when a single client repeatedly requests access to the inference service at a rate higher than the configured threshold: \n \n This is just one of the many security features to protect our inference service. Click here to find out more on Securing Model Serving in Red Hat OpenShift AI (on ROSA) with F5 Distributed Cloud API Security. \n Demonstration \n In a real-world scenario, the front-end application could be hosted on the cloud, or hosted at the edge, or served through F5 Distributed Cloud, offering flexible alternatives for efficient application delivery based on user preferences and specific needs. To illustrate how all the discussed components work seamlessly together, we simplify our example by deploying Open WebUI as the front-end application on the Red Hat OpenShift cluster in the data center in California, which includes RAG services. While a DPU or GPU could be used for improved performance, our setup utilizes a CPU for inferencing tasks. \n We connect our app to our enterprise data stores deployed on NetApp ONTAP in the data center in Seattle using F5 Distributed Cloud Network Connect, where we have a copy of \"Chapter 1. About the Migration Toolkit for Virtualization\" from Red Hat. These documents are processed and saved to the Vector DB: \n \n Our embedding Large Language Model (LLM) is Sentence-Transformers/all-MiniLM-L6-v2, and here is our RAG template: \n \n Instead of connecting to the inference endpoint on Red Hat OpenShift AI (RHOAI) on ROSA directly, we connect to the F5 Distributed Cloud HTTP Load Balancer (rhoai-llm-serving.f5-demo.com) from F5 Distributed Cloud App Connect: \n \n Previously, we asked, \"What is MTV?“ and we never received a response related to Red Hat Migration Toolkit for Virtualization: \n \n Now, let's try asking the same question again with RAG services enabled: \n \n We finally received the response we had anticipated. \n Next, we use F5 Distributed Cloud Web App & API Protection to safeguard our Red Hat OpenShift AI (RHOAI) inference endpoint on ROSA by rate-limiting the access, thus preventing a single client from exhausting the inference service: \n \n As expected, we received \"Too Many Requests\" in the response on our app upon requesting the inference service at a rate greater than the set threshold: \n \n With F5 Distributed Cloud's real-time observability and security analytics from the F5 Distributed Console, we can proactively monitor for potential threats. For example, if necessary, we can block a client from accessing the inference service by adding it to the Blocked Clients List: \n \n As expected, this specific client is now unable to access the inference service: \n \n Summary \n Deploying and securing RAG for enterprise RAG-enabled AI applications in a multi-vendor, hybrid, and multi-cloud environment can present complex challenges. In collaboration with Red Hat OpenShift AI (RHOAI) and NetApp ONTAP, F5 Distributed Cloud provides an effortless solution that secures RAG components seamlessly for enterprise RAG-enabled AI applications. ","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})@stringLength":"10043","kudosSumWeight":1,"repliesCount":0,"readOnly":false,"images":{"__typename":"AssociatedImageConnection","edges":[{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDE","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtWVU1amF6?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDI","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtYXEzVmNL?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDM","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtSmJNZ3Ry?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDQ","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtSVJiZ3dK?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDU","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtTHA2SDc0?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDY","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtdTFySzhB?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDc","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtN1dLVFhZ?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDg","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtS3RzbXJl?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDk","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtc0l2OTBl?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDEw","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtNGlPcnZQ?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDEx","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtbUtVeVYz?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDEy","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtRVBsOVg3?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDEz","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtdUJkT0V0?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDE0","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtd2YxMEQ3?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDE1","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtZDF2R25C?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDE2","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtdGV0Sk53?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDE3","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtZjdZOEU3?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDE4","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtV2xodXRO?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDE5","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtS2ZzOW5O?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDIw","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQta2xHUGlO?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDIx","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtbmp0UFR4?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDIy","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtZU05eG1o?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDIz","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtWE1saWtw?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDI0","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtTllHYk9Z?revision=25\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDI1","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzk4ODQtclVkT3E3?revision=25\"}"}}],"totalCount":28,"pageInfo":{"__typename":"PageInfo","hasNextPage":true,"endCursor":"MjUuMnwyLjF8b3wyNXxfTlZffDI1","hasPreviousPage":false,"startCursor":null}},"videos":{"__typename":"VideoConnection","edges":[],"totalCount":0,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}}},"Conversation:conversation:310629":{"__typename":"Conversation","id":"conversation:310629","topic":{"__typename":"TkbTopicMessage","uid":310629},"lastPostingActivityTime":"2025-02-20T02:52:06.099-08:00","solved":false},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMTA2MjktMjM4NTJpNjU0MjJBMjFFODAxMjUwRQ?revision=44\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMTA2MjktMjM4NTJpNjU0MjJBMjFFODAxMjUwRQ?revision=44","title":"DC-Cover_0031_sergiu-valena-zpqqUshRHWY-unsplash.jpg","associationType":"COVER","width":500,"height":500,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMTA2MjktMjI5MDFpMTFFQkU1NjM3M0EyMEM2Rg?revision=44\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMTA2MjktMjI5MDFpMTFFQkU1NjM3M0EyMEM2Rg?revision=44","title":"CNF software focused diagram.png","associationType":"BODY","width":2556,"height":2085,"altText":null},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMTA2MjktMjI1OTZpQzQwNUYxQUU4REVBRTVBRQ?revision=44\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMTA2MjktMjI1OTZpQzQwNUYxQUU4REVBRTVBRQ?revision=44","title":"Overall-CNF-architecture.png","associationType":"BODY","width":2626,"height":1164,"altText":null},"TkbTopicMessage:message:310629":{"__typename":"TkbTopicMessage","subject":"Introducing F5 BIG-IP Next CNF Solutions for Red Hat OpenShift","conversation":{"__ref":"Conversation:conversation:310629"},"id":"message:310629","revisionNum":44,"uid":310629,"depth":0,"board":{"__ref":"Tkb:board:TechnicalArticles"},"author":{"__ref":"User:user:303102"},"teaser@stripHtml({\"removeProcessingText\":true,\"truncateLength\":-1})":"","introduction":"","metrics":{"__typename":"MessageMetrics","views":2265},"postTime":"2023-05-24T05:00:00.035-07:00","lastPublishTime":"2023-06-05T21:33:25.493-07:00","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})":" \n \n 5G and Red Hat OpenShift \n 5G standards have embraced Cloud-Native Network Functions (CNFs) for implementing network services in software as containers. This is a big change from previous Virtual Network Functions (VNFs) or Physical Network Functions (PNFs). \n The main characteristics of Cloud-Native Functions are: \n \n Implementation as containerized microservices \n Small performance footprint, with the ability to scale horizontally \n Independence of guest operating system, since CNFs operate as containers \n Lifecycle manageable by Kubernetes \n \n Overall, these provide a huge improvement in terms of flexibility, faster service delivery, resiliency, and crucially using Kubernetes as unified orchestration layer. The later is a drastic change from previous standards where each vendor had its own orchestration. This unification around Kubernetes greatly simplifies network functions for operators, reducing cost of deploying and maintaining networks. Additionally, by embracing the container form factor, allows Network Functions (NFs) to be deployed in new use cases like far edge. This is thanks to the smaller footprint while at the same time these can be also deployed at large scale in a central data center because of the horizontal scalability. \n In this article we focus on Red Hat OpenShift which is the market leading and industry reference implementation of Kubernetes for IT and Telco workloads. \n Introduction to F5 BIG-IP Next CNF Solutions \n F5 BIG-IP Next CNF Solutions is a suite of Kubernetes native 5G Network Functions, implemented as microservices. It shares the same Cloud Native Engine (CNE) as F5 BIG-IP Next SPK introduced last year. The functionalities implemented by the CNF Solutions deal mainly with user plane data. \n User plane data has the particularity that the final destination of the traffic is not the Kubernetes cluster but rather an external end-point, typically the Internet. In other words, the traffic gets in the Kubernetes cluster and it is forwarded out of the cluster again. This is done using dedicated interfaces that are not used for the regular ingress and egress paths of the regular traffic of a Kubernetes cluster. In this case, the main purpose of using Kubernetes is to make use of its orchestration, flexibility, and scalability. \n The main functionalities implemented at initial GA release of the CNF Solutions are: \n \n F5 Next Edge Firewall CNF, an IPv4/IPv6 firewall with the main focus in protecting the 5G core networks from external threads, including DDoS flood protection and IPS DNS protocol inspection. \n F5 Next CGNAT CNF, which offers large scale NAT with the following features:\n \n NAPT, Port Block Allocation, Static NAT, Address Pooling Paired, and Endpoint Independent mapping modes. \n Inbound NAT and Hairpining. \n Egress path filtering and address exclusions. \n ALG support: FTP/FTPS, TFTP, RTSP and PPTP. \n \n \n F5 Next DNS CNF, which offers a transparent DNS resolver and caching services. Other remarkable features are:\n \n Zero rating \n DNS64 which allows IPv6-only clients connect to IPv4-only services via synthetic IPv6 addresses. \n \n \n F5 Next Policy Enforcer CNF, which provides traffic classification, steering and shaping, and TCP and video optimization. This product is launched as Early Access in February 2023 with basic functionalities. Static TCP optimization is now GA in the initial release. \n \n Although the CGNAT (Carrier Grade NAT) and the Policy Enforcer functionalities are specific to User Plane use cases, the Edge Firewall and DNS functionalities have additional uses in other places of the network. \n F5 and OpenShift \n BIG-IP Next CNF Solutions fully supports Red Hat OpenShift Container Platform which allows the deployment in edge or core locations with a unified management across the multiple deployments. OpenShift operators greatly facilitates the setup and tuning of telco grade applications. These are: \n \n Node Tuning Operator, used to setup Hugepages. \n CPU Manager and Topology Manager with NUMA awareness which allows to schedule the data plane PODs within a NUMA domain which is aligned with the SR-IOV NICs they are attached to. \n \n In an OpenShift platform all these are setup transparently to the applications and BIG-IP Next CNF Solutions uniquely require to be configured with an appropriate runtimeClass. \n F5 BIG-IP Next CNF Solutions architecture \n F5 BIG-IP Next CNF Solutions makes use of the widely trusted F5 BIG-IP Traffic Management Microkernel (TMM) data plane. This allows for a high performance, dependable product from the start. The CNF functionalities come from a microservices re-architecture of the broadly used F5 BIG-IP VNFs. The below diagram illustrates how a microservices architecture used. \n \n The data plane POD scales vertically from 1 to 16 cores and scales horizontally from 1 to 32 PODs, enabling it to handle millions of subscribers. NUMA nodes are supported. \n The next diagram focuses on the data plane handling which is the most relevant aspect for this CNF suite: \n Typically, each data plane POD has two IP address, one for each side of the N6 reference point. These could be named radio and Internet sides as shown in the diagram above. \n The left-side L3 hop must distribute the traffic amongst the lef-side addresses of the CNF data plane. This left-side L3 hop can be a router with BGP ECMP (Equal Cost Multi Path), an SDN or any other mechanism which is able to: \n \n Distribute the subscribers across the data plane PODs, shown in [1] of the figure above. \n Keep these subscribers in the same PODs when there is a change in the number of active data plane PODs (scale-in, scale-out, maintenance, etc...) as shown in [2] in the figure above. This minimizes service disruption. \n \n In the right side of the CNFs, the path towards the Internet, it is typical to implement NAT functionality to transform telco's private addresses to public addresses. This is done with the BIG-IP Next CG-NAT CNF. This NAT makes the return traffic symmetrical by reaching the same POD which processed the outbound traffic. This is thanks to each POD owning part of this NAT space, as shown in [3] of the above figure. Each POD´s NAT address space can be advertised via BGP. \n When not using NAT in the right side of the CNFs, it is required that the network is able to send the return traffic back to the same POD which is processing the same connection. The traffic must be kept symmetrical at all times, this is typically done with an SDN. \n Using F5 BIG-IP Next CNF Solutions \n As expected in a fully integrated Kubernetes solution, both the installation and configuration is done using the Kubernetes APIs. The installation is performed using helm charts, and the configuration using Custom Resource Definitions (CRDs). Unlike using ConfigMaps, using CRDs allow for schema validation of the configurations before these are applied. Details of the CRDs can be found in this clouddocs site. Next it is shown an overview of the most relevant CRDs. \n General network configuration \n Deploying in Kubernetes automatically configures and assigns IP addresses to the CNF PODs. The data plane interfaces will require specific configuration. The required steps are: \n \n Create Kubernetes NetworkNodePolicies and NetworkAttchment definitions which will allow to expose SR-IOV VFs to the CNF data planes PODs (TMM). To make use of these SR-IOV VFs these are referenced in the BIG-IP controller's Helm chart values file. This is described in the Networking Overview page. \n Define the L2 and L3 configuration of the exposed SR-IOV interfaces using the F5BigNetVlan CRD. \n If static routes need to be configured, these can be added using the F5BigNetStaticroute CRD. \n If BGP configuration needs to be added, this is configured in the BIG-IP controller's Helm chart values file. This is described in the BGP Overview page. It is expected this will be configured using a CRD in the future. \n \n Traffic management listener configuration \n As with classic BIG-IP, once the CNFs are running and plumbed in the network, no traffic is processed by default. The traffic management functionalities implemented by BIG-IP Next CNF Solutions are the same of the analogous modules in the classic BIG-IP, and the CRDs in BIG-IP Next to configure these functionalities are conceptually similar too. \n Analogous to Virtual Servers in classic BIG-IP, BIG-IP Next CNF Solutions have a set of CRDs that create listeners of traffic where traffic management policies are applied. This is mainly the F5BigContextSecure CRD which allows to specify traffic selectors indicating VLANs, source, destination prefixes and ports where we want the policies to be applied. \n There are specific CRDs for listeners of Application Level Gateways (ALGs) and protocol specific solutions. These required several steps in classic BIG-IP: first creating the Virtual Service, then creating the profile and finally applying it to the Virtual Server. In BIG-IP Next this is done in a single CRD. At time of this writing, these CRDs are: \n \n F5BigZeroratingPolicy - Part of Zero-Rating DNS solution; enabling subscribers to bypass rate limits. \n F5BigDnsApp - High-performance DNS resolution, caching, and DNS64 translations. \n F5BigAlgFtp - File Transfer Protocol (FTP) application layer gateway services. \n F5BigAlgTftp - Trivial File Transfer Protocol (TFTP) application layer gateway services. \n F5BigAlgPptp - Point-to-Point Tunnelling Protocol (PPTP) application layer gateway services. \n F5BigAlgRtsp - Real Time Streaming Protocol (RTSP) application layer gateway services. \n \n Traffic management profiles and policies configuration \n Depending on the type of listener created, these can have attached different types of profiles and policies. \n In the case of F5BigContextSecure it can get attached the following CRDs to define how traffic is processed: \n \n F5BigTcpSetting - TCP options to fine-tune how application traffic is managed. \n F5BigUdpSetting - UDP options to fine-tune how application traffic is managed. \n F5BigFastl4Setting - FastL4 option to fine-tune how application traffic is managed. \n \n and the following policies for security and NAT: \n \n F5BigDdosPolicy - Denial of Service (DoS/DDoS) event detection and mitigation. \n F5BigFwPolicy - Granular stateful-flow filtering based on access control list (ACL) policies. \n F5BigIpsPolicy - Intelligent packet inspection protects applications from malignant network traffic. \n F5BigNatPolicy - Carrier-grade NAT (CG-NAT) using large-scale NAT (LSN) pools. \n \n The ALG listeners require the use of F5BigNatPolicy and might make use for the F5BigFwPolicy CRDs. These CRDs have also traffic selectors to allow further control over which traffic these policies should be applied to. \n Firewall Contexts \n Firewall policies are applied to the listener with best match. In addition to the F5BigFwPolicy that might be attached, a global firewall policy (hence effective in all listeners) can be configured before the listener specific firewall policy is evaluated. This is done with F5BigContextGlobal CRD, which can have attached a F5BigFwPolicy. F5BigContextGlobal also contains the default action to apply on traffic not matching any firewall rule in any context (e.g. Global Context or Secure Context or another listener). This default action can be set to accept, reject or drop and whether to log this default action. In summary, within a listener match, the firewall contexts are processed in this order: \n \n ContextGlobal \n Matching ContextSecure or another listener context. \n Default action as defined by ContextGlobal's default action. \n \n Event Logging \n Event logging at high speed is critical to provide visibility of what the CNFs are doing. For this the next CRDs are implemented: \n \n F5BigLogProfile - Specifies subscriber connection information sent to remote logging servers. \n F5BigLogHslpub - Defines remote logging server endpoints for the F5BigLogProfile. \n \n Demo \n \n \n F5 BIG-IP Next CNF Solutions roadmap \n What it is being exposed here is just the begin of a journey. Telcos have embraced Kubernetes as compute and orchestration layer. Because of this, BIG-IP Next CNF Solutions will eventually replace the analogous classic BIG-IP VNFs. Expect in the upcoming months that BIG-IP Next CNF Solutions will match and eventually surpass the features currently being offered by the analogous VNFs. \n Conclusion \n \n \n \n This article introduces fully re-architected, scalable solution for Red Hat OpenShift mainly focused on telco's user plane. This new microservices architecture offers flexibility, faster service delivery, resiliency and crucially the use of Kubernetes. Kubernetes is becoming the unified orchestration layer for telcos, simplifying infrastructure lifecycle, and reducing costs. OpenShift represents the best-in-class Kubernetes platform thanks to its enterprise readiness and Telco specific features. The architecture of this solution alongside the use of OpenShift also extends network services use cases to the edge by allowing the deployment of Network Functions in a smaller footprint. Please check the official BIG-IP Next CNF Solutions documentation for more technical details and check www.f5.com for a high level overview. \n \n \n \n \n ","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})@stringLength":"13334","kudosSumWeight":3,"repliesCount":3,"readOnly":false,"images":{"__typename":"AssociatedImageConnection","edges":[{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDE","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMTA2MjktMjM4NTJpNjU0MjJBMjFFODAxMjUwRQ?revision=44\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDI","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMTA2MjktMjI5MDFpMTFFQkU1NjM3M0EyMEM2Rg?revision=44\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDM","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMTA2MjktMjI1OTZpQzQwNUYxQUU4REVBRTVBRQ?revision=44\"}"}}],"totalCount":3,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}},"videos":{"__typename":"VideoConnection","edges":[{"__typename":"VideoEdge","cursor":"MHxodHRwczovL3lvdXR1LmJlL25HRXVuUW5VYjBRfDB8MjU7MjV8fA","node":{"__typename":"AssociatedVideo","videoTag":{"__typename":"VideoTag","vid":"https://youtu.be/nGEunQnUb0Q","thumbnail":"https://i.ytimg.com/vi/nGEunQnUb0Q/hqdefault.jpg","uploading":false,"height":225,"width":400,"title":null},"videoAssociationType":"INLINE_BODY"}}],"totalCount":1,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}}},"Conversation:conversation:338353":{"__typename":"Conversation","id":"conversation:338353","topic":{"__typename":"TkbTopicMessage","uid":338353},"lastPostingActivityTime":"2025-01-02T16:09:26.863-08:00","solved":false},"User:user:326558":{"__typename":"User","uid":326558,"login":"Eric_Ji","registrationData":{"__typename":"RegistrationData","status":null},"deleted":false,"avatar":{"__typename":"UserAvatar","url":"https://community.f5.com/t5/s/zihoc95639/images/dS0zMjY1NTgtaDBOSjVu?image-coordinates=0%2C0%2C1110%2C1110"},"id":"user:326558"},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtTXJvZ2o4?revision=60\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtTXJvZ2o4?revision=60","title":"image.png","associationType":"BODY","width":2325,"height":1011,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtOE9BVFdX?revision=60\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtOE9BVFdX?revision=60","title":"image.png","associationType":"BODY","width":2412,"height":1386,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtWDZYT0JG?revision=60\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtWDZYT0JG?revision=60","title":"image.png","associationType":"BODY","width":2013,"height":1210,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtZGV6OEJ4?revision=60\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtZGV6OEJ4?revision=60","title":"image.png","associationType":"BODY","width":1722,"height":1102,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtd0t4N0FC?revision=60\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtd0t4N0FC?revision=60","title":"image.png","associationType":"BODY","width":2564,"height":1630,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtdGZMc1lQ?revision=60\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtdGZMc1lQ?revision=60","title":"image.png","associationType":"BODY","width":1726,"height":1134,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtb3lGSkxH?revision=60\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtb3lGSkxH?revision=60","title":"image.png","associationType":"BODY","width":857,"height":965,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMteHpyb0kx?revision=60\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMteHpyb0kx?revision=60","title":"image.png","associationType":"BODY","width":1379,"height":620,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtYUVPSVJS?revision=60\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtYUVPSVJS?revision=60","title":"image (3).png","associationType":"BODY","width":1348,"height":636,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtY1NqeDJj?revision=60\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtY1NqeDJj?revision=60","title":"image (4).png","associationType":"BODY","width":540,"height":435,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtTjAxZG1E?revision=60\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtTjAxZG1E?revision=60","title":"image.png","associationType":"BODY","width":2004,"height":1054,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMteWRUcTVk?revision=60\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMteWRUcTVk?revision=60","title":"image.png","associationType":"BODY","width":3410,"height":2222,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtQzBuYmhU?revision=60\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtQzBuYmhU?revision=60","title":"image.png","associationType":"BODY","width":858,"height":650,"altText":""},"TkbTopicMessage:message:338353":{"__typename":"TkbTopicMessage","subject":"Securing Model Serving in Red Hat OpenShift AI (on ROSA) with F5 Distributed Cloud API Security","conversation":{"__ref":"Conversation:conversation:338353"},"id":"message:338353","revisionNum":60,"uid":338353,"depth":0,"board":{"__ref":"Tkb:board:TechnicalArticles"},"author":{"__ref":"User:user:326558"},"teaser@stripHtml({\"removeProcessingText\":true,\"truncateLength\":-1})":" Learn how Red Hat OpenShift AI on ROSA and F5 Distributed Cloud API Security work together to protect generative AI model inference endpoints. This integration ensures robust API discovery, schema enforcement, LLM-aware threat detection, bot mitigation, sensitive data redaction, and continuous observability—enabling secure, compliant, and high-performance AI-driven experiences at scale. ","introduction":"","metrics":{"__typename":"MessageMetrics","views":778},"postTime":"2024-12-18T12:01:43.571-08:00","lastPublishTime":"2024-12-18T12:01:43.571-08:00","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})":" As enterprises embrace Generative AI—particularly deploying large language models (LLMs) and other foundational AI models—production environments become increasingly complex. Organizations need an end-to-end MLOps platform that streamlines the entire lifecycle: developing, training, fine-tuning, and especially serving models at scale. \n Red Hat OpenShift AI (OpenShift AI) meets this need by providing a comprehensive, hybrid MLOps environment. When deployed on Red Hat OpenShift Service on AWS (ROSA), OpenShift AI takes advantage of AWS’s managed infrastructure services and cloud-native elasticity, enabling organizations to scale AI/ML workflows efficiently and cost-effectively. \n Yet, as models are served to end-users or integrated into downstream applications, security considerations become paramount. Inference endpoints may be targeted for unauthorized access, data exfiltration, or prompt manipulation. F5 Distributed Cloud addresses these challenges by offering robust capabilities—API discovery, schema enforcement, LLM-aware threat detection, bot mitigation, sensitive data redaction, and continuous observability—to ensure that inference endpoints remain secure, compliant, and high-performing. \n In this post, we will: \n \n Introduce OpenShift AI and its capabilities for AI/ML workloads with an emphasis on model serving. \n Discuss how running OpenShift AI on ROSA leverages AWS services for scalable, cost-effective AI/ML operations. \n Show how F5 Distributed Cloud API Security enhances the security posture of generative AI model inference endpoints—covering automated API discovery, schema enforcement, threat detection, rate limiting, and compliance measures. \n Demonstrate how to integrate these capabilities end-to-end, using Ollama Mistral-7B as an example model. \n \n What is Red Hat OpenShift AI for Generative AI Applications? \n Red Hat OpenShift AI is a hybrid MLOps platform designed to simplify the entire AI lifecycle. It empowers teams to train, fine-tune, deploy, serve, monitor, and continuously improve generative AI models. By merging IT operations, data science workflows, and application development practices into one platform, OpenShift AI accelerates innovation, fosters governance, and encourages collaboration—essential ingredients for delivering enterprise-grade AI solutions. \n Key Features: \n \n Hybrid MLOps Platform: Unified environment for data scientists, developers, and operators, whether you run on-premises, in the cloud, or in a hybrid setup. \n Distributed Workloads & Fine-Tuning: Scale model training and fine-tuning across distributed compute frameworks, adapting large language models to your domain’s requirements. \n Model Serving & Monitoring: Deploy and serve models at scale using technologies like KServe, ModelMesh, and a variety of specialized runtimes (OpenVINO™ Model Server, Caikit/TGIS, NVIDIA Triton Inference Server, etc.). Continuously monitor model performance, detect drift, and ensure ongoing model quality. \n Lifecycle Management & DevOps Integration: Seamlessly integrate data science pipelines with CI/CD workflows. Automate model deployment, rollout new versions safely, and achieve consistent delivery of AI-driven features. \n Enhanced Collaboration: Enable data scientists, developers, and IT Ops to work together using notebooks (JupyterLab), popular frameworks (TensorFlow, PyTorch), and unified governance, speeding up innovation and time-to-value. \n \n High-Level Architecture: OpenShift AI integrates workbenches, distributed workloads, data science pipelines, serving engines, and monitoring tools atop Kubernetes and OpenShift operators, leveraging GitOps, pipelines, service mesh, and serverless technologies. \n \n Figure 1: OpenShift AI High-Level Architecture \n Running OpenShift AI on ROSA for Scalable AI/ML Solutions \n Red Hat OpenShift Service on AWS (ROSA) brings a fully managed OpenShift environment to AWS. This allows teams to focus on building and serving AI models rather than managing infrastructure. \n Key advantages: \n \n Scalability: Seamlessly scale GPU-accelerated compute, storage, and networking resources as model serving workloads grow. \n Cost Efficiency & On-Demand Resources: Leverage Amazon EC2 instances, Amazon S3, and other AWS services only as needed, paying as you go. \n Unified Management: Offload cluster operations and lifecycle management to ROSA, ensuring reliable operations and freeing your team to concentrate on AI innovation. \n \n Security Challenges in Model Serving for Generative AI \n While OpenShift AI and ROSA simplify operations, serving AI models still raises critical security concerns: \n \n Unauthorized Access & Data Leakage: External requests to inference endpoints may attempt to extract proprietary knowledge or sensitive data from the model. \n Prompt Injection & Malicious Content: LLMs can be tricked into producing harmful or confidential outputs if the prompts are manipulated. \n Bot Attacks & Performance Risks: Automated scripts can overwhelm inference endpoints, degrade performance, or escalate costs. \n Compliance & Sensitive Data Handling: AI outputs can contain PII or regulated data, necessitating encryption, redaction, and audit trails to meet compliance demands. \n Evolving Threat Landscape: The complexity and dynamism of AI models and APIs call for continuous posture management and adaptive threat detection. \n \n Enhancing Security with F5 Distributed Cloud API Security \n F5 Distributed Cloud provides a comprehensive set of capabilities tailored for securing modern AI inference endpoints. By integrating these capabilities with OpenShift AI deployments on ROSA, organizations gain: \n \n Automated API Discovery & Posture Management:\n \n Identify all inference endpoints automatically, eliminating hidden APIs. \n Enforce schemas based on observed traffic to ensure requests and responses follow expected patterns. \n Integrate “shift-left” security checks into CI/CD pipelines, catching misconfigurations before production. \n \n \n LLM-Aware Threat Detection & Request Validation:\n \n Detect attempts to manipulate prompts or break compliance rules, ensuring suspicious requests are blocked before reaching the model. \n \n \n Bot Mitigation & Adaptive Rate Controls:\n \n Differentiate between legitimate users and bots or scrapers, blocking automated attacks. \n Dynamically adjust rate limits and policies based on usage history and real-time conditions, maintaining performance and reliability. \n \n \n Sensitive Data Redaction & Compliance:\n \n Identify and mask PII or sensitive tokens in requests and responses. \n Adhere to data protection regulations and maintain detailed logs for auditing, monitoring, and compliance reporting. \n \n \n Seamless Integration & Observability:\n \n Deploy F5 Distributed Cloud seamlessly alongside OpenShift AI on ROSA without reshuffling existing architecture. \n Use centralized dashboards and analytics to monitor real-time metrics—latency, error rates, compliance indicators—to continuously refine policies and adapt to emerging threats. \n \n \n \n Example: Working with Multiple OLLAMA Models and Programmatic Inference \n In this scenario, multiple OLLAMA models have been deployed on the OpenShift cluster. For instance: \n sh-5.1$ ollama ls \nNAME ID SIZE MODIFIED \nllama2:7b 78e26419b446 3.8 GB 3 days ago \nllama3.2:1b baf6a787fdff 1.3 GB 2 weeks ago \nmario:latest 7434c42677ab 3.8 GB 12 seconds ago \nmistral:latest f974a74358d6 4.1 GB 11 days ago \norca-mini:latest 2dbd9f439647 2.0 GB About a minute ago\nphi3:latest 4f2222927938 2.2 GB 2 weeks ago \nphi3:mini 4f2222927938 2.2 GB 2 weeks ago \ntinyllama:latest 2644915ede35 637 MB 2 weeks ago \n \n We have a variety of models—ranging from Mistral-7B to tinyllama—that can be served simultaneously. While the environment is currently using CPUs for hosting these models, you could leverage GPUs for better performance in a production scenario. \n Unlike the model training and fine-tuning phase—where we worked directly within OpenShift AI Workbenches (to be covered in future blogs)— in this scenario we’re querying the LLM endpoint from a local application, where Python libraries and LangChain are installed. Instead of hitting the cluster directly, we route traffic through an F5 Distributed Cloud-managed URL (e.g., http://llm01.volt.thebizdevops.net). In a real-world deployment, the frontend could also be hosted on OpenShift or served through F5 Distributed Cloud Regional Edges (RE), providing flexible options for scaling and delivering the application. \n This ensures that all requests pass through F5 Distributed Cloud’s security layers, applying policies, detecting threats, and protecting sensitive data before they reach the LLM endpoint hosted in OpenShift AI on ROSA. \n \n Figure 2: Integrated Architecture with F5 Distributed Cloud and OpenShift AI on AWS \n F5 Distributed Cloud Capabilities in Action \n To illustrate the key F5 Distributed Cloud features, we’ve divided them into distinct capabilities and included screen captures in the F5 Distributed Cloud console. These captures provide a visual reference to understand how each capability enhances the security and compliance posture of your LLM inference endpoints. \n 1. API Discovery & Schema Enforcement \n What It Does: F5 Distributed Cloud automatically identifies all exposed inference endpoints for your AI/LLM models. It then derives schemas from real traffic, enforcing expected request and response formats. By blocking malformed inputs early, your model stays protected, ensuring consistent, reliable, and trustworthy inferences. \n \n Figure 3: API Discovery & Schema Enforcement (Refer to the annotated image above showing multiple discovered endpoints, shadow APIs, and downloadable OpenAPI specifications derived from actual traffic patterns.) \n 2. LLM-Aware Threat Detection & Request Validation \n What It Does: This feature identifies potential threats to your LLM endpoints by enforcing strict OpenAPI-based validation on incoming requests. By catching invalid or suspicious inputs early, you can adjust policies to block them in the future, ensuring that malicious attempts—whether aiming to exploit the LLM’s behavior or break compliance rules—never reach the inference logic. \n \n Figure 4: Security Analysis for a Non-Compliant Request (Here, the request triggered an OpenAPI validation failure. Although currently “allowed,” policies can be easily configured to “block” these events going forward, preventing non-compliant or potentially harmful requests from impacting your LLM models.) \n \n \n Figure 5: API Inventory Validation Configuration (This image illustrates the corresponding configuration settings for OpenAPI Validation. By validating both requests and responses at multiple layers—headers, body, and content-type—F5 Distributed Cloud ensures that LLM prompts remain safe, compliant, and free from injection attacks.) \n 3. Bot Mitigation & Rate Limiting \n What It Does: Differentiates legitimate user traffic from automated bots or scrapers bot mitigation, and ensures fair usage of resources (rate limiting). By differentiating legitimate requests from bot-driven abuse and enforcing request thresholds, F5 Distributed Cloud protects inference endpoints from performance degradation while maintaining a positive user experience. \n \n Figure 6: Bot Defense in Action (This image shows how F5 Distributed Cloud identifies and classifies automated traffic as “Bad Bots,” blocking them to preserve endpoint availability and prevent resource exhaustion.) \n \n \n Figure 7: Configuring Rate Limits (This image illustrates the setup of request thresholds for a specific endpoint, ensuring no single client overwhelms the inference service.) \n \n \n Figure 8: Enforced Rate Limit (429 Too Many Requests) (This image demonstrates a client exceeding the above-configured request limit and receiving a 429 response, confirming that F5 Distributed Cloud’s rate limiting is actively maintaining fair resource allocation.) \n 4. Sensitive Data Redaction & Compliance Logging \n What It Does: Identifies and masks personally identifiable information (PII) or other sensitive data—such as credit card numbers, emails, and phone numbers—within model responses. New Sensitive Data Exposure Rules allow you to customize and enforce policies to block or redact sensitive fields dynamically. This ensures compliance with frameworks like HIPAA, GDPR., and other regulatory mandates while capturing detailed logs for auditing. \n \n Figure 9: Adding Sensitive Data Exposure Rules (This image shows how you can add custom rules to detect and control exposure of sensitive fields, such as card-expiration dates, phone numbers, and credit-card details, ensuring model responses comply with organizational security policies.) \n \n \n Figure 10: Sensitive Data Detection Across APIs (Here, sensitive data types—like social-security numbers and phone numbers—are automatically detected across API responses. Built-in and custom rules flag potential exposures, empowering teams to enforce redaction and maintain compliance.) \n \n \n Figure 11: Service Policy for Model Validation (This image shows a service policy in action, blocking an inference request that doesn’t meet defined model validation criteria. Such policies can also be tied to compliance mandates, ensuring non-compliant responses are never returned to clients.) \n \n \n Figure 12: API Compliance & Sensitive Data Detection (Here, sensitive fields like credit-card or phone-number are automatically identified, and associated compliance frameworks (HIPAA, GDPR) are recognized. This empowers you to enforce data redaction, maintain regulatory compliance, and produce audit-ready logs without revealing sensitive details.) \n 5. Centralized Observability & Continuous Policy Updates \n What It Does: Offers dashboards and analytics tools to monitor request volumes, latency, errors, and compliance metrics across your AI inference endpoints. Security teams can leverage these observations to continuously refine their policies, enhance schema validations, and recalibrate rate limits as threats evolve or model usage grows. \n \n Figure 13: Endpoint-Level Metrics Dashboard (This example shows an LLM endpoint /api/generate with available metrics including error rate, latency, request rate, request size, response size, and throughput. By monitoring these trends, teams can quickly identify performance bottlenecks, detect anomalies, and apply targeted policy changes to maintain optimal efficiency and security.) \n The Outcome: Secure, Compliant, and Performant LLM Serving \n By combining Red Hat OpenShift AI on ROSA with F5 Distributed Cloud, organizations can: \n \n Confidently serve multiple LLMs at scale, handling diverse use cases and workloads. \n Securely expose inference endpoints, ensuring that requests from external clients are validated, sanitized, and protected against prompt injection, unauthorized access, or excessive traffic. \n Maintain compliance and privacy, redacting sensitive data and logging requests for auditing and reporting purposes. \n Continuously adapt to evolving threats, leveraging real-time observability and agile policy management for persistent security improvements. \n \n This powerful combination enables generative AI models to be woven into complex enterprise workflows—such as insurance claims processing—without sacrificing trust, governance, or user satisfaction. \n Conclusion \n Red Hat OpenShift AI on ROSA, bolstered by F5 Distributed Cloud API Security, provides a robust, scalable, and secure foundation for running generative AI workloads in production. Together, they address the nuanced security challenges of exposing LLM inference endpoints to external clients. \n Whether you are working with a single LLM or managing a portfolio of models like OLLAMA’s Mistral, Phi3, and TinyLlama, this combination ensures that your users—connecting from anywhere—can trust the quality, security, and compliance of the AI services they rely on. ","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})@stringLength":"16330","kudosSumWeight":5,"repliesCount":4,"readOnly":false,"images":{"__typename":"AssociatedImageConnection","edges":[{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDE","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtTXJvZ2o4?revision=60\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDI","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtOE9BVFdX?revision=60\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDM","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtWDZYT0JG?revision=60\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDQ","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtZGV6OEJ4?revision=60\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDU","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtd0t4N0FC?revision=60\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDY","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtdGZMc1lQ?revision=60\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDc","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtb3lGSkxH?revision=60\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDg","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMteHpyb0kx?revision=60\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDk","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtYUVPSVJS?revision=60\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDEw","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtY1NqeDJj?revision=60\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDEx","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtTjAxZG1E?revision=60\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDEy","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMteWRUcTVk?revision=60\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDEz","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMzgzNTMtQzBuYmhU?revision=60\"}"}}],"totalCount":13,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}},"videos":{"__typename":"VideoConnection","edges":[],"totalCount":0,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}}},"Conversation:conversation:329457":{"__typename":"Conversation","id":"conversation:329457","topic":{"__typename":"TkbTopicMessage","uid":329457},"lastPostingActivityTime":"2024-05-06T05:00:00.024-07:00","solved":false},"User:user:423130":{"__typename":"User","uid":423130,"login":"Ivan_Hurtt","registrationData":{"__typename":"RegistrationData","status":null},"deleted":false,"avatar":{"__typename":"UserAvatar","url":"https://community.f5.com/t5/s/zihoc95639/m_assets/avatars/custom/Frankenstack_11-1706132273780.svg?time=1706132301000"},"id":"user:423130"},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjk0NTcteUtpcE01?revision=2\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjk0NTcteUtpcE01?revision=2","title":"clipboard_image-1-1714517590562.png","associationType":"BODY","width":1432,"height":793,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjk0NTctR1g0cWxC?revision=2\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjk0NTctR1g0cWxC?revision=2","title":"clipboard_image-2-1714517590562.png","associationType":"BODY","width":1984,"height":858,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjk0NTctM3QyOXFF?revision=2\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjk0NTctM3QyOXFF?revision=2","title":"clipboard_image-3-1714517590562.png","associationType":"BODY","width":1970,"height":848,"altText":""},"TkbTopicMessage:message:329457":{"__typename":"TkbTopicMessage","subject":"Maximizing Scalability with F5 CIS: Blue-Green Deployment on OpenShift","conversation":{"__ref":"Conversation:conversation:329457"},"id":"message:329457","revisionNum":2,"uid":329457,"depth":0,"board":{"__ref":"Tkb:board:TechnicalArticles"},"author":{"__ref":"User:user:423130"},"teaser@stripHtml({\"removeProcessingText\":true,\"truncateLength\":-1})":"","introduction":"This video demonstrates how F5 CIS can maximize scalability, providing a per-application, blue-green strategy using an OpenShift multi-cluster deployment","metrics":{"__typename":"MessageMetrics","views":228},"postTime":"2024-05-06T05:00:00.024-07:00","lastPublishTime":"2024-05-06T05:00:00.024-07:00","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})":" The following video tutorials and Github repo will help you learn how F5 Container Ingress Services (CIS) can integrate with Red Hat Advanced Cluster Management (ACM) to maximize scalability and provide a per-application blue-green strategy in an OpenShift multi-cluster environment. \n In this 20 Minute demo you will get a detailed understanding of: \n \n Redhat ACM running in a management cluster fetching KubeConfigs from the ConfigMap for worker clusters (Coffee Pod & Tea Pod) \n F5 CIS subscribing to KubeConfig and listening for events from worker clusters (Coffee Pod & Tea Pod) \n F5 CIS posting AS3 declaration to BIG-IP. BIG-IP executing all necessary commands to achieve desired end state (ie increase cluster resources to 6 instances per pod) \n \n Demo on YouTube video \n \n \n \n \n GitHub Repo repo \n Notes \n If you want to make sure the service doesn’t take any share of the overall traffic distribution, set the Route base service weight to “0”. Although the pool for the base service gets created (without any pool members for this case), the traffic is distributed among the extended services only. \n \n \n \n \n One more point to note is that in the extendedConfigmap mode is set to “ratio”. This is because currently CIS processes the weights only in two cases \n \n The route is an A/B route \n mode is set to ratio \n \n \n \n \n \n \n However, the cluster ratio doesn't play any role here as we have not defined any cluster ratio values for any of the clusters. So, each cluster will have equal weightage. Therefore, in this case, traffic distribution will be solely based on service weights. (which is the requirement for this case) ","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})@stringLength":"1746","kudosSumWeight":1,"repliesCount":0,"readOnly":false,"images":{"__typename":"AssociatedImageConnection","edges":[{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDE","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjk0NTcteUtpcE01?revision=2\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDI","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjk0NTctR1g0cWxC?revision=2\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDM","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjk0NTctM3QyOXFF?revision=2\"}"}}],"totalCount":3,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}},"videos":{"__typename":"VideoConnection","edges":[],"totalCount":0,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}}},"Conversation:conversation:327094":{"__typename":"Conversation","id":"conversation:327094","topic":{"__typename":"TkbTopicMessage","uid":327094},"lastPostingActivityTime":"2024-04-22T05:00:00.029-07:00","solved":false},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtWmp1akVW?revision=20\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtWmp1akVW?revision=20","title":"UC1.png","associationType":"BODY","width":2572,"height":1590,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtRnk0b3dQ?revision=20\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtRnk0b3dQ?revision=20","title":"UC2.png","associationType":"BODY","width":3664,"height":1782,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtNWNXY0Vt?revision=20\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtNWNXY0Vt?revision=20","title":"UC3.png","associationType":"BODY","width":3756,"height":1944,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtM2JBV3JZ?revision=20\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtM2JBV3JZ?revision=20","title":"UC4.png","associationType":"BODY","width":2548,"height":350,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtaURaU1do?revision=20\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtaURaU1do?revision=20","title":"UC5.png","associationType":"BODY","width":3012,"height":1702,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtU3B2bUVn?revision=20\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtU3B2bUVn?revision=20","title":"UC6.png","associationType":"BODY","width":1756,"height":428,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtTzhBOEha?revision=20\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtTzhBOEha?revision=20","title":"UC7.png","associationType":"BODY","width":862,"height":92,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtRE96Q3o1?revision=20\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtRE96Q3o1?revision=20","title":"UC8.png","associationType":"BODY","width":2488,"height":188,"altText":""},"TkbTopicMessage:message:327094":{"__typename":"TkbTopicMessage","subject":"F5 BIG-IP deployment with OpenShift - per-application 2-tier deployments","conversation":{"__ref":"Conversation:conversation:327094"},"id":"message:327094","revisionNum":20,"uid":327094,"depth":0,"board":{"__ref":"Tkb:board:TechnicalArticles"},"author":{"__ref":"User:user:303102"},"teaser@stripHtml({\"removeProcessingText\":true,\"truncateLength\":-1})":"","introduction":"","metrics":{"__typename":"MessageMetrics","views":902},"postTime":"2024-04-22T05:00:00.029-07:00","lastPublishTime":"2024-04-22T05:00:00.029-07:00","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})":" Introduction \n This article dives deeper in 2-tier deployments introduced in the previous article F5 BIG-IP deployment with OpenShift - platform and networking options yet we will start comparing a common Kubernetes/OpenShift architecture with a more capable and flexible architecture using BIG-IP. Although this type of deployment can be done with the BIG-IP in Layer 4 (L4) mode this article will focus in using the BIG-IP in Layer 7 (L7) mode which allows for more functionality and flexibility. A regular Kubernetes/OpenShift deployment is as follows: \n \n Typically, the External LB doesn´t add much added value and just spreads the connections across the ingress controller instances without any service intelligence. This article emphasizes on the added value of F5 BIG-IP with Container Ingress Services (CIS) by operating at L7. Additionally, it is described how CIS is ingress agnostic and can be used at is fullest regardless of the ingress controller, API manager, service mesh used, or even a combination of these! This 2-tier architecture can be seen next, where it is not only shown that traffic is sent directly to the ingress controllers without the NodePort indirection. In the figure, it can also be seen compared to a 1-tier architecture: \n \n \n ClusterIP and NodePort \n When using a 2-tier deployment, a key aspect that needs to be decided is how it is going to be sent the traffic to the ingress controllers. \n NodePort \n With other External LB solutions (ELB), the common way to send traffic to the ingress controllers in Kubernetes is by exposing these as NodePort. This option always works because the ELB doesn't need to deal with the CNI. In this case, when the ELB sends traffic to a node, the NodePort construct by default load balances the traffic to any node in the cluster where the ingress controller is. This means that there would be 3 hops before reaching the workload POD. These hops are: the ELB, the ingress controller, the NodePort, and finally the workload POD. This NodePort load balancing to other nodes can be removed by changing the externalTrafficPolicy from Cluster to Local in the Service definition. Thus reducing the number of hops from 3 to 2. \n ClusterIP \n Alternatively, the BIG-IP can send the traffic directly to the POD addresses, aka ClusterIP mode, without any translation or Kubernetes construct in between the BIG-IP and the ingress controller. This is specified in CIS with the --pool-member-type=cluster option and requires support of the CNI. Using OpenShiftSDN and ClusterIP is discouraged because this CNI is being deprecated and will not be covered in this article. When using OVNKubernetes, the BIG-IP sends the traffic without any tunnelling. At this moment, this requires that the clusters are in the same subnet as the BIG-IP. Which option to choose? \n \n If the OpenShift cluster is not in the same subnet as the BIG-IP or if the infrastructure doesn´t allow POD addresses traverse the segment (ie: because of a L2 firewall/micro-segmentation), then NodePort would need to be used. \n I would discourage the use of NodePort with externalTrafficPolicy: Cluster because it doesn´t allow to do adequate health check monitoring because each probe will be sent to a different ingress controller. Moreover, persistence and troubleshooting is more tricky too because of the additional hop that NodePort creates. If NodePort needs to be used, then externalTrafficPolicy: Local referencing the ingress controllers is recommended. \n If there is no problem in routing the traffic to the OpenShift nodes using the POD addresses, I would encourage using ClusterIP because it is a transparent method, where the application traffic is easier to track: there are no additional hops or address translations. \n \n OpenShift´s bundled ingress/router is HA-proxy; In on-prem OpenShift deployments, the default router instance is created using hostNetwork access. HostNetwork can also be used with CIS --pool-member-type=cluster mode. The IP addresses of the pool members will be the node addresses, but there will not be any indirection to the POD. \n The examples provided in the git repository mentioned in this article use ClusterIP but can be easily modified to NodePort if necessary. \n 2-tier per-service load balancing and health checking \n This type of deployment requires that the external load balancer, in this case the F5 BIG-IP, can take actions based on the L7 route path. This includes health-checking the application and performing TLS/SSL termination. From a Kubernetes/OpenShift point of view, the most remarkable is that it is needed to expose the L7 routes twice. That is, for a given L7 route there will be a manifest for the ingress controller and another L7 route manifest for CIS. This is outlined in the next figure: \n \n In the above figure, there is only one ingress controller instance with several replicas serving the same L7 routes. It is not relevant which ingress controller it is. This ingress controller can have defined its L7 routes with any resource type. This is transparent from the point of view of CIS. CIS only requires that the ingress controller can be referenced with a selector from a Service. This service is shown in brown. There will be a Service manifest for each L7 route for CIS. Even if all these services point to the same ingress controller instances. This allows you to: \n \n Monitor each L7 route´s backends individually. \n Have a clean NetOps/DevOps separation where DevOps can manipulate the ingress controller configurations freely and NetOps can control the publishing of these in the ELB. \n Ultimately make load-balancing decisions in a per-application basis \n \n The L7 routes for CIS can be defined either as VirtualServer, Route (as NextGen Route) or as Ingress resource types. Using the Ingress resource type is not recommended because of its limited functionalities compared with the other types. Overall, there will be a 1:1:1 mapping between the L7 routes defined for CIS, the Service manifests referenced by CIS´ L7 routes, and the L7 routes defined for the ingress controller. This is shown conceptually in the next figure. \n \n With respect to load balancing algorithm, given that the number of ingress controllers in a two-tier deployment is typically small it is sensible to take into account the load of each ingress controller by using an algorithm such as least-connections, least-sessions (if using persistence) or fastest (application response time) which should send more traffic to the ingress controller which is performing better or in other words, would avoid sending more traffic to an ingress controller that is not performing well. \n Unify all ingresses in a single VIP \n Thanks to CIS being ingress controller agnostic, it is also possible to combine L7 routes from different ingress controllers, these can be a mix of one or more ingress controllers, including service mesh ingress or API managers. This brings the flexibility of exposing L7 routes from these sources in a single VIP, as shown in the next figure. \n \n From the picture above, it can be seen how in this arrangement the ingress controllers are not limited to a single L7 route. Also note that as mentioned previously, it is needed to have a 1:1 mapping between the L7 routes defined in the BIG-IP (1st tier) and in the in-cluster ingress element (2nd tier). By default, these paired L7 routes are expected to match their URL. If necessary, it is possible to do URL translation by using the hostRewrite and rewrite attributes in the VirtualServer CR. \n OpenShift´s documentation mentions several use cases for creating additional OpenShift router (HA-proxy) instances, referred to as route sharding. These are: \n \n Balance Ingress Controllers, or routers, with several routes to speed up responses to changes. \n Allocate certain routes to have different reliability guarantees than other routes. \n Allow certain Ingress Controllers to have different policies defined. \n Allow only specific routes to use additional features. \n Expose different routes on different addresses so that internal and external users can see different routes, for example. \n Transfer traffic from one version of an application to another during a blue-green deployment. \n \n An additional use case is to isolate the control-plane Routes of the cluster and the application Routes in independent HA-proxy instances. This way it is not only it can be guaranteed different levels of reliability but also guaranteeing resource allocation for the cluster´s own traffic. \n In the section \"Using additional Router instances\" section below, it is shown how to do these configurations. \n Monitoring in a 2-tier deployment \n For each L7 route, regardless of the resource type used (Ingress, Route or VirtualServer), these share the same schema for monitoring. \n \n Next are shown the relevant parameters, where send is the key parameter that specifies L7 route: \n monitor:\n type: http\n send: \"GET / HTTP/1.1\\r\\nHost: www.twotier.com\\r\\nConnection: close\\r\\n\\r\\n\"\n recv: \"^HTTP/1.1 200\" \n In many cases, https will be used instead. HA-proxy requires that the health monitor uses TLS 1.2. By default, the https monitor defaults to TLS 1.0, hence we will need to specify one of the BIG-IP default TLS profiles as shown next: \n monitor:\n type: https\n send: \"GET / HTTP/1.1\\r\\nHost: www.twotier.com\\r\\nConnection: close\\r\\n\\r\\n\"\n recv: \"^HTTP/1.1 200\"\n sslProfile: /Common/serverssl \n When using NGINX or Istio ingress gateway, using TLS SNI (Server Name Indication) is requierd. In cases where SNI is required, the following needs to be done: \n \n One time, in TMM monitoring needs to be enabled as indicated in https://my.f5.com/manage/s/article/K85210713 \n For each FQDN it is needed to pre-create a server-side TLS profile in the partition /Common specifying the server name used as TLS SNI. \n \n \n The resulting monitor section will typically look as follows: \n monitor:\n type: https\n send: \"GET / HTTP/1.1\\r\\nHost: www.twotier.com\\r\\nConnection: close\\r\\n\\r\\n\"\n recv: \"^HTTP/1.1 200\"\n sslProfile: /Common/serverssl-www.twotier.com \n Note that this TLS profile is only needed for the monitoring, for application´s traffic the BIG-IP forwards the SNI received in the VIP. \n Alternative monitoring in a 2-tier deployment \n In this article, it is preferred to have 1:1 mapping between the L7 routes defined in the BIG-IP and the L7 routes defined in the ingress controller. It is possible to reduce the number of L7 routes and Service manifests for CIS by adding several monitors to a single L7 route manifest. For example we could have a single base L7 route such as www.twotier.com/ and several monitors probing the different applications behind this FQDN, for example: www.twotier.com/, www.twotier.com/account, and www.twotier.com/shop. This alternative monitoring can be configured with the following parameters: \n \n In this example, you can see how to create such a configuration. The preference for using a 1:1 mapping instead is because although this alternative strategy creates less manifests, automating the creation of these might be more complex. Additionally, it will be less easy to know in a given moment which applications are not working fine. \n Using OpenShift´s Router (HA-Proxy) in the second tier \n CIS can be configured to ingest F5 CRs, OpenShift Routes CRs or Ingress resources. When CIS is configured for using Routes or Ingress, by default HA-Proxy will also ingest these. In other words, both CIS and HA-Proxy will implement the same L7 routes. This behavior also applies to single-tier deployments where the behavior could be desired for testing. In general this behavior is not desired. HA-Proxy can be configured to do-not evaluate Routes or Ingress manifests with a given label or to do-not search for these in namespaces with a given label, for example \"router=bigip\". This behavior is configured in the IngressController resource type that manages HA-proxy's configuration: \n {\n \"spec\": {\n \"namespaceSelector\": {\n \"matchExpressions\": [\n {\n \"key\": \"router\",\n \"operator\": \"NotIn\",\n \"values\": [\n \"bigip\"\n ]\n }\n ]\n }\n }\n} \n This JSON configuration is applied with the next command for the default ingress controller/router in OpenShift: \n oc -n openshift-ingress-operator patch ingresscontroller default --patch-file=router-default.shard.json --type=merge \n More information on Route sharding can be found in this OpenShift document. \n Using the default Router instance \n OpenShift´s installer deploys the default HA-proxy instance with endpointPublishingStrategy type set to either hostNetwork or LoadBalancer. This will depend whether the OpenShift deployment is on-prem, in a public cloud, etc... This can be checked with the following command: \n oc get ingresscontrollers default -n openshift-ingress-operator -o jsonpath='{.status.endpointPublishingStrategy}{\"\\n\"}' \n Regardless HA-proxy is configured with endpointPublishingStrategy either hostNetwork or LoadBalancer, CIS can be configured in clusterIP mode and refer to the HA-Proxy PODs using the \"--orchestration-cni=ovn-k8s\" and \"--pool-member-type=cluster\" options. The Service to refer to these HA-Proxy PODs will look as follows: \n apiVersion: v1\nkind: Service\nmetadata:\nname: router-default-route-a\nnamespace: openshift-ingress\nspec:\nports:\n- name: http\n port: 80\n protocol: TCP\n targetPort: http\n- name: https\n port: 443\n protocol: TCP\n targetPort: https\nselector:\n ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default\n type: ClusterIP \n Where as indicated earlier there will be a Service manifest for each L7 route in order to have per L7 route monitoring. Note that these Services reside in the openshift-ingress namespace, where the HA-Proxy PODs are. Examples of this configuration can be found here using F5 CRs. \n Using additional Router instances \n Creating a new HA-Proxy instance just requires creating a new IngressController manifest in the openshift-ingress-operator namespace, such as the following sample: \n apiVersion: operator.openshift.io/v1\nkind: IngressController\nmetadata:\nname: shard-apps\nnamespace: openshift-ingress-operator\nspec:\nclientTLS:\ndefaultCertificate:\n name: router-shard-secret\ndomain: shard-apps.com\nendpointPublishingStrategy:\ntype: Private\nnamespaceSelector:\n matchExpressions:\n - key: router\n operator: In\n values:\n - shard-apps\n replicas: 1 \n From this sample, note that: \n \n This new Router instance will evaluate only the L7 routes with namespaces with the label \"router=shard-apps\". \n the endpointPublishingStrategy is set to type Private, this is because we don´t need further exposure of it when CIS is accessing it in ClusterIP mode. \n \n When using an additional HA-proxy instance, the default Router will need to be instructed not to evaluate these same L7 routes as the additional HA-proxy instance. This is shown next: \n {\n \"spec\": {\n \"namespaceSelector\": {\n \"matchExpressions\": [\n {\n \"key\": \"router\",\n \"operator\": \"NotIn\",\n \"values\": [\n \"bigip\",\n \"shard-apps\"\n ]\n }\n ]\n }\n }\n} \n An example of this configuration can be found here using Routes and multiple shards. \n Using Istio Ingress Gateway / OpenShift Service Mesh \n Using Istio (aka Service Mesh as packaged in OpenShift) in the second tier is straight-forward because Istio and CIS use different resource types for configuring L7 routes. A sample Service manifest to point to Istio ingress gateway is shown next \n apiVersion: v1\nkind: Service\nmetadata:\n name: svc-route-a\nnamespace: istio-system\nspec:\nports:\n- name: https\n port: 8443\n protocol: TCP\n targetPort: https\nselector:\n app: istio-ingressgateway\ntype: ClusterIP \n Where it can be observed that the Service manifests need to be created in the istio-system namespace and the selector needs to match the Istio ingress gateway´s Deployment's label. An example of this configuration can be found here using F5 CRs. \n Using NGINX \n Using NGINX in the second tier is much like using Istio, next is shown a sample Service manifest pointing to NGINX instances: \n apiVersion: v1\nkind: Service\nmetadata:\nname: svc-route-a\nnamespace: nginx-ingress\nspec:\nports:\n- name: https\n port: 443\n protocol: TCP\n targetPort: 443\nselector:\n app.kubernetes.io/instance: nginxingress-sample\n app.kubernetes.io/name: nginx-ingress\n type: ClusterIP \n The only special consideration to be considered, and only if both NGINX and CIS are meant to use the Ingress resource type, is to specify in the Ingress manifests which controller will ingest the manifest. This is specified with the ingressClassName attribute as shown next: \n in the case of NGINX \n apiVersion: networking.k8s.io/v1 \nkind: Ingress\n[...]\nspec:\n ingressClassName: nginx \n or in the case of CIS \n spec:\n ingressClassName: f5 \n An example of this configuration using F5 CRs in CIS and Ingress in NGINX can be found \n Using advanced BIG-IP services with CIS \n BIG-IP has many advanced services that can be easily referenced using F5 VirtualServer CRs, either directly in the manifest or referencing a Policy CR. When using Next Gen Routes these can also reference Policy CR but in a per router-group basis, hence having somewhat less granularity. Next is an overview of some of these advanced functionalities available. \n \n \n VirtualServer CR attributes for configuring advanced services \n \n \n \n Functionality \n attribute \n \n \n Advanced Web Application Firewall \n waf \n \n \n L3 anti-DoS protection \n dos \n \n \n L7 bot defense \n botDefense \n \n \n HTML content rewrite \n htmlProfile \n \n \n Custom iRules to perform advanced traffic management \n iRules \n \n \n Message routing framework (MRF) functionality for non HTTP traffic \n httpMrfRoutingEnabled \n \n \n IP intelligence \n ipIntelligencePolicy \n \n \n L3 Firewall rules \n firewallPolicy \n \n \n \n \n The configuration of these services must be pre-created in the BIG-IP as profiles in the /Common partition which can be referenced freely by CIS. It is expected that CIS 2.17 will incorporate APM and SSLo. \n Conclusion and closing remarks \n F5 BIG-IP is an External Loadbalancer with application awareness which allows to do advanced L7 services (not just TLS termination) and unify the different ingress paths into the cluster, allowing to merge several ingress controllers, API managers, service mesh or a combination of these. This solution is ingress type agnostic. \n A two-tier arrangement allows clean DevOps and NetOps separation. This is done by having separate L7 routes defined for the in-cluster ingress controllers and for the BIG-IPs. \n To continue your journey, please check the examples in this GitHub repository using VirtualServer F5 CRs and OpenShift Route CRs. ","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})@stringLength":"19099","kudosSumWeight":0,"repliesCount":0,"readOnly":false,"images":{"__typename":"AssociatedImageConnection","edges":[{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDE","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtWmp1akVW?revision=20\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDI","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtRnk0b3dQ?revision=20\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDM","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtNWNXY0Vt?revision=20\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDQ","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtM2JBV3JZ?revision=20\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDU","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtaURaU1do?revision=20\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDY","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtU3B2bUVn?revision=20\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDc","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtTzhBOEha?revision=20\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDg","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjcwOTQtRE96Q3o1?revision=20\"}"}}],"totalCount":8,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}},"videos":{"__typename":"VideoConnection","edges":[],"totalCount":0,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}}},"Conversation:conversation:328268":{"__typename":"Conversation","id":"conversation:328268","topic":{"__typename":"TkbTopicMessage","uid":328268},"lastPostingActivityTime":"2024-03-28T10:12:30.816-07:00","solved":false},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjgyNjgtVTlPYWdG?revision=9\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjgyNjgtVTlPYWdG?revision=9","title":"Screenshot 2024-03-06 at 10.07.35.png","associationType":"BODY","width":2354,"height":1510,"altText":""},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjgyNjgtaGNwVlVZ?revision=9\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjgyNjgtaGNwVlVZ?revision=9","title":"Screenshot 2024-03-06 at 10.26.50.png","associationType":"BODY","width":3828,"height":2128,"altText":""},"TkbTopicMessage:message:328268":{"__typename":"TkbTopicMessage","subject":"F5 BIG-IP per application Red Hat OpenShift cluster migrations","conversation":{"__ref":"Conversation:conversation:328268"},"id":"message:328268","revisionNum":9,"uid":328268,"depth":0,"board":{"__ref":"Tkb:board:TechnicalArticles"},"author":{"__ref":"User:user:303102"},"teaser@stripHtml({\"removeProcessingText\":true,\"truncateLength\":-1})":"","introduction":"Perform cluster migrations, with no service disruption and avoiding all-or-nothing changes","metrics":{"__typename":"MessageMetrics","views":491},"postTime":"2024-03-18T11:25:13.068-07:00","lastPublishTime":"2024-03-28T10:12:30.816-07:00","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})":" Overview \n OpenShift migrations are typically done when it is desired to minimise disruption time when performing cluster upgrades. Disruptions can especially occur when performing big changes in the cluster such as changing the CNI from OpenShiftSDN to OVNKubernetes. OpenShift cluster migrations are well covered for applications by using RedHat's Migration Toolkit for Containers (MTC). The F5 BIG-IP has the role of network redirector indicated in the Network considerations chapter. The F5 BIG-IP can perform per L7 route migration without service disruption, hence allowing migration or roll-back on a per-application basis, eliminating disruption and de-risking the maintenance window. \n How it works \n As mentioned above, the traffic redirection will be done on a per L7 route basis, this is true regardless of how these L7 routes are implemented: ingress controller, API manager, service mesh, or a combination of these. This L7 awareness is achieved by using F5 BIG-IP's Controller Ingress Services (CIS) controller for Kubernetes/OpenShift and its multi-cluster functionality which can expose in a single VIP L7 routes of services hosted in multiple Kubernetes/OpenShift clusters. This is shown in the next picture. \n \n For a migration operation it will be used a blue/green strategy independent for each L7 route where blue will refer to the application in the older cluster and green will refer to the application in the newer cluster. For each L7 route, it will be specified a weight for each blue or green backend (like in an A/B strategy). This is shown in the next picture. \n \n In this example, the migration scenario uses OpenShift´s default ingress controller (HA proxy) as an in-cluster ingress controller where the Route CR is used to indicate the L7 routes. For each L7 route defined in the HA-proxy tier, it will be defined as an L7 route in the F5 BIG-IP tier. This 1:1 mapping allows to have the per-application granularity. The VirtualServer CR is used for the F5 BIG-IP. If desired, it is also possible to use Route resources for the F5 BIG-IP. Next, it is shown the manifests for a given L7 route required for the F5 BIG-IP, in this case, https://www.migration.com/shop (alias route-b) \n apiVersion: \"cis.f5.com/v1\" kind: VirtualServer metadata: name: route-b namespace: openshift-ingress labels: f5cr: \"true\" spec: host: www.migration.com virtualServerAddress: \"10.1.10.106\" hostGroup: migration.com tlsProfileName: reencrypt-tls profileMultiplex: \"/Common/oneconnect-32\" pools: - path: /shop service: router-default-route-b-ocp1 servicePort: 443 weight: 100 alternateBackends: - service: router-default-route-b-ocp2 weight: 0 monitor: type: https name: /Common/www.migration.com-shop reference: bigip --- apiVersion: v1 kind: Service metadata: annotations: name: router-default-route-b-ocp1 namespace: openshift-ingress spec: ports: - name: http port: 80 protocol: TCP targetPort: http - name: https port: 443 protocol: TCP targetPort: https selector: ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default type: NodePort --- apiVersion: v1 kind: Service metadata: annotations: name: router-default-route-b-ocp2 namespace: openshift-ingress spec: ports: - name: http port: 80 protocol: TCP targetPort: http - name: https port: 443 protocol: TCP targetPort: https selector: ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default type: NodePort\n \n The CIS multi-cluster feature will search for the specified services in both clusters. It is up to DevOps to ensure that blue services are only present in the cluster designated as blue (in this case router-default-route-b-ocp1) and green services are only present in the cluster designated as green (in this case router-default-route-b-ocp2). It is important to remark that the Route manifests for HA-proxy (or any other ingress solution used) doesn't require any modification. That is, this migration mechanism is transparent to the application developers. \n Demo \n You can see this feature in action in the next video \n \n The manifests used in the demo can be found in the following GitHub repository: \n https://github.com/f5devcentral/f5-bd-cis-demo/tree/main/crds/demo-mc-twotier-haproxy-noshards \n Next steps \n \n Try it today! CIS is open-source software and is included in your support entitlement. If you want to learn more about CIS and CIS multi-cluster features the following blog articles are suggested. \n \n F5 BIG-IP deployment with OpenShift - platform and networking options \n F5 BIG-IP deployment with OpenShift - publishing application options \n F5 BIG-IP deployment with OpenShift - multi-cluster architectures \n ","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})@stringLength":"5091","kudosSumWeight":0,"repliesCount":0,"readOnly":false,"images":{"__typename":"AssociatedImageConnection","edges":[{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDE","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjgyNjgtVTlPYWdG?revision=9\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDI","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjgyNjgtaGNwVlVZ?revision=9\"}"}}],"totalCount":2,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}},"videos":{"__typename":"VideoConnection","edges":[],"totalCount":0,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}}},"Conversation:conversation:328325":{"__typename":"Conversation","id":"conversation:328325","topic":{"__typename":"TkbTopicMessage","uid":328325},"lastPostingActivityTime":"2024-03-25T05:00:00.040-07:00","solved":false},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjgzMjUtNWhpazhN?revision=14\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjgzMjUtNWhpazhN?revision=14","title":"Screenshot 2024-03-07 at 14.36.02.png","associationType":"BODY","width":3330,"height":2154,"altText":""},"TkbTopicMessage:message:328325":{"__typename":"TkbTopicMessage","subject":"F5 BIG-IP deployment with Red Hat OpenShift - the no CIS option","conversation":{"__ref":"Conversation:conversation:328325"},"id":"message:328325","revisionNum":14,"uid":328325,"depth":0,"board":{"__ref":"Tkb:board:TechnicalArticles"},"author":{"__ref":"User:user:303102"},"teaser@stripHtml({\"removeProcessingText\":true,\"truncateLength\":-1})":" This article is a set of recommendations when using BIG-IP as external load balancer of an OpenShift cluster with the default router (HA-proxy) and not using CIS ","introduction":"This article is a set of recommendations when using F5 BIG-IP as external load balancer of an Red Hat OpenShift cluster with the default router (HA-proxy) and not using CIS","metrics":{"__typename":"MessageMetrics","views":1702},"postTime":"2024-03-25T05:00:00.040-07:00","lastPublishTime":"2024-03-25T05:00:00.040-07:00","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})":" Overview \n There are deployments where F5 BIG-IP is used in front of a Red Hat OpenShift cluster without the Container Ingress Services (CIS) controller. In general, this is not recommended because without CIS there is no Kubernetes integration and automation. This article is for deployments where for some reason CIS is not used. The article assumes the default OpenShift deployment where HA-proxy (aka OpenShift´s router) is used for ingress where ports 80 and 443 are exposed in separate pools. \n Implementation \n The next is a set of recommendations for appropriate health checking and failure handling of HA-proxy instances in OpenShift: \n \n Create an HTTP monitor for the readiness endpoint When a POD is not in Ready state CIS automatically removes it from the pool. When not using CIS we need to mimic this functionality with an HTTP monitor in port 1936 (readiness endpoint) for both 443 and 80 pools: \n ltm monitor http ocp_haproxy {\n destination *.1936\n recv \"^HTTP/1.1 200\"\n send \"GET /healthz/ready HTTP/1.1\\r\\nHost: localhost\\r\\nConnection: close\\r\\n\\r\\n\"\n}\n \n This monitor will detect the following failure scenarios: \n \n There is no IP connectivity to the HA-proxy instance \n HA-proxy is not working properly (it will return a non-HTTP 200 status code) \n HA-proxy is performing a graceful shutdown (a POD deletion), for example when doing an upgrade (returning an HTTP 500 status code). \n \n The graceful shutdown process of HA-proxy is as follows: \n \n the readiness endpoint immediately returns an HTTP 500 error but will continue processing requests for the applications for 45 seconds. This is indicated in the response of the readiness endpoint with an ”[+]backend-http ok” message in the payload. \n after 45 seconds, the endpoint will return “[-]backend-http failed: reason withheld” and will TCP RESET any request for the applications. Create an HTTP monitor probing the readiness endpoint (port 1936) of HA-proxy. During these 45 seconds, the timer values to the HTTP monitor have plenty of time to disable the HA-proxy instance appropriately. \n \n \n \n \n Handle nongraceful errors with the pool's reselect tries To deal better with nongraceful shutdowns or transient errors, we can make use of this mechanism which will reselect a new HA-proxy instance when a request to an application fails. The recommendation is to set the number of tries to the number of HA-proxy instances -1. \n \n \n Set an additional TCP monitor for HA-proxy´s application's path socket This additional TCP monitor in either port 80 or 443 complements the HTTP monitor of the readiness endpoint by additionally validating that the HA-proxy instances can listen for requests in their designated application's socket. Although this is handled with the reselect tries mechanism this monitor will provide visibility that such types of errors are happening. \n \n \n Established connections For already established connections the Action on Service Down feature could be used but it is considered that the default option NONE should be used. This will allow us to potentially finish already established connections. Using RESET would trigger a faster retry from the HTTP client with the disadvantage of not allowing finishing requests that could potentially finish. Other options are not worth considering. \n \n Final configuration \n In the next image, it can be seen graphically the configuration recommended with key settings highlighted. \n \n Conclusions \n F5 BIG-IP has sophisticated features for health checking and handling failures in backend services, including Kubernetes, to operate without any service disruption. Although in this article we configured BIG-IP for OpenShift manually, it is recommended to use the CIS controller which brings OpenShift visibility, automation more advanced use cases to BIG-IP. CIS is open-source software and is included in your support contract entitlement. The next links provide a good introduction to CIS: \n \n F5 BIG-IP deployment with OpenShift - platform and networking options \n F5 BIG-IP deployment with OpenShift - publishing application options \n \n F5 BIG-IP deployment with OpenShift - multi-cluster architectures \n \n ","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})@stringLength":"4245","kudosSumWeight":0,"repliesCount":0,"readOnly":false,"images":{"__typename":"AssociatedImageConnection","edges":[{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDE","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjgzMjUtNWhpazhN?revision=14\"}"}}],"totalCount":1,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}},"videos":{"__typename":"VideoConnection","edges":[],"totalCount":0,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}}},"Conversation:conversation:325774":{"__typename":"Conversation","id":"conversation:325774","topic":{"__typename":"TkbTopicMessage","uid":325774},"lastPostingActivityTime":"2024-02-20T09:35:27.785-08:00","solved":false},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY4OTJpMkE5OTM0QkM5NDU5QTIwQg?revision=15\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY4OTJpMkE5OTM0QkM5NDU5QTIwQg?revision=15","title":"Screenshot 2023-12-16 at 16.05.34.png","associationType":"BODY","width":1966,"height":1194,"altText":"Screenshot 2023-12-16 at 16.05.34.png"},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MzRpMEJENTc2RTZFRTY3Qjc0Mg?revision=15\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MzRpMEJENTc2RTZFRTY3Qjc0Mg?revision=15","title":"Screenshot 2023-12-18 at 15.58.18.png","associationType":"BODY","width":1702,"height":1080,"altText":"Screenshot 2023-12-18 at 15.58.18.png"},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MzVpQ0YyMzc2NkYxMkRDOUQzNA?revision=15\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MzVpQ0YyMzc2NkYxMkRDOUQzNA?revision=15","title":"Screenshot 2023-12-18 at 16.11.40.png","associationType":"BODY","width":2232,"height":1340,"altText":"Screenshot 2023-12-18 at 16.11.40.png"},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MjJpMzQ2QzNDOTc0NUMzODhEOQ?revision=15\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MjJpMzQ2QzNDOTc0NUMzODhEOQ?revision=15","title":"Screenshot 2023-12-18 at 12.52.16.png","associationType":"BODY","width":2364,"height":1462,"altText":"Screenshot 2023-12-18 at 12.52.16.png"},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MjNpMDk0ODMyNTczMEQ3RDMwNw?revision=15\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MjNpMDk0ODMyNTczMEQ3RDMwNw?revision=15","title":"Screenshot 2023-12-18 at 12.52.32.png","associationType":"BODY","width":2366,"height":1462,"altText":"Screenshot 2023-12-18 at 12.52.32.png"},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MjVpNzA1MkFDODgzRDNCQjc3NA?revision=15\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MjVpNzA1MkFDODgzRDNCQjc3NA?revision=15","title":"Screenshot 2023-12-18 at 13.10.30.png","associationType":"BODY","width":2358,"height":1452,"altText":"Screenshot 2023-12-18 at 13.10.30.png"},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MzFpRUM5NzdBN0U3MDIyQkFCNA?revision=15\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MzFpRUM5NzdBN0U3MDIyQkFCNA?revision=15","title":"Screenshot 2023-12-18 at 13.15.51.png","associationType":"BODY","width":2000,"height":1224,"altText":"Screenshot 2023-12-18 at 13.15.51.png"},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MThpMzNBNDU1REI3MDc0QzM2MA?revision=15\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MThpMzNBNDU1REI3MDc0QzM2MA?revision=15","title":"Screenshot 2023-12-18 at 11.40.27.png","associationType":"BODY","width":2154,"height":1190,"altText":"Screenshot 2023-12-18 at 11.40.27.png"},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MTlpNkU3RTBEMzk3MzEwNzhDRA?revision=15\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MTlpNkU3RTBEMzk3MzEwNzhDRA?revision=15","title":"Screenshot 2023-12-18 at 11.41.39.png","associationType":"BODY","width":2164,"height":1210,"altText":"Screenshot 2023-12-18 at 11.41.39.png"},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MjBpOUExQzEyQTZEQzI4Q0M1MA?revision=15\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MjBpOUExQzEyQTZEQzI4Q0M1MA?revision=15","title":"Screenshot 2023-12-18 at 12.44.56.png","associationType":"BODY","width":1856,"height":1602,"altText":"Screenshot 2023-12-18 at 12.44.56.png"},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5NDNpNzg4NzExOTg3NUIxM0M5Qw?revision=15\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5NDNpNzg4NzExOTg3NUIxM0M5Qw?revision=15","title":"Screenshot 2023-12-19 at 12.42.03.png","associationType":"BODY","width":2158,"height":1600,"altText":"Screenshot 2023-12-19 at 12.42.03.png"},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5NDRpQkVGRDNDQkZERkZDNTgxNQ?revision=15\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5NDRpQkVGRDNDQkZERkZDNTgxNQ?revision=15","title":"Screenshot 2023-12-19 at 12.59.40.png","associationType":"BODY","width":2756,"height":1368,"altText":"Screenshot 2023-12-19 at 12.59.40.png"},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5NDZpNEMyRTJCMDM0Q0VFMzQyQQ?revision=15\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5NDZpNEMyRTJCMDM0Q0VFMzQyQQ?revision=15","title":"Screenshot 2023-12-19 at 14.57.04.png","associationType":"BODY","width":2274,"height":1450,"altText":"Screenshot 2023-12-19 at 14.57.04.png"},"TkbTopicMessage:message:325774":{"__typename":"TkbTopicMessage","subject":"F5 BIG-IP deployment with OpenShift - multi-cluster architectures","conversation":{"__ref":"Conversation:conversation:325774"},"id":"message:325774","revisionNum":15,"uid":325774,"depth":0,"board":{"__ref":"Tkb:board:TechnicalArticles"},"author":{"__ref":"User:user:303102"},"teaser@stripHtml({\"removeProcessingText\":true,\"truncateLength\":-1})":" This functionality enables to load balance services that are spread across multiple clusters. It is important to remark that this functionality is Service oriented: the load balancing decisions are independent for each Service. ","introduction":"","metrics":{"__typename":"MessageMetrics","views":1221},"postTime":"2024-01-22T05:00:00.019-08:00","lastPublishTime":"2024-02-20T09:35:27.785-08:00","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})":" \n This is a continuation of the previous articles on OpenShift and BIG-IP: \n \n F5 BIG-IP deployment with OpenShift - platform and networking options \n F5 BIG-IP deployment with OpenShift - publishing application options \n \n Introduction \n During last quarter of 2023, multi-cluster support has been added to Container Ingress Services (CIS). This functionality enables to load balance services that are spread across multiple clusters. It is important to remark that this functionality is Service oriented: the load balancing decisions are independent for each Service. Moreover, the different clusters don´t have to be mirror copies of each other, applications can be spread heterogeneously across the clusters. These clusters can also have different versions of OpenShift. This is shown in the next picture \n Please also note that although the picture might suggest a 1-tier deployment, this functionality can be used in either 1-tier or 2-tier deployments, in both cases in a Layer 7/application/service oriented manner. You can watch an overview of this functionality in the next video \n \n \n Scenarios and use cases \n This CIS multi-cluster feature is meant to be used in the following scenarios: \n \n multiple clusters within the same data centre \n multiple clusters in stretched data centers \n multiple clusters in different Availability Zones of the same cloud region \n \n This multi-cluster feature allows to expose in a single VIP services from multiple clusters regardless where they are hosted, this allows: \n \n OpenShift migrations (threfore this applies to single cluster deployments too) \n Increase capacity \n Increase applications availability, ie: during OpenShift platform maintenance windows \n Blue-green (A/B testing) across clusters \n Route sharding across clusters (spreading the applications across the clusters arbitrarily) \n Split large clusters into smaller ones \n \n Per-service multi-cluster architecture \n In the Kubernetes ecosystem, many external load balancers just load balance across the different Ingress Controllers instances without any application awareness. In the case of BIG-IP this is not the case. BIG-IP is Kubernetes Service aware and can make load balancing decisons (and apply BIG-IP functionalities) in a per application basis, depending on the L7 route. This BIG-IP and CIS functionality also applies when using CIS in a multi-cluster deployment. This allows applications to be spread across the different clusters in an heterogeneous manner. The BIG-IP can health check them individually. \n It is also worth to remark that the CIS multi-cluster functionality is: \n \n agnostic whether the deployment uses a 1-tier or 2-tier approach \n independent of the ingress controller(s) used and individual L7 routes can always be health checked \n \n In the next picture it is shown how CIS is deployed. We can see in the picture that there are two CIS instances running in different clusters for redundancy purposes. \n Notice in the picture that clusters other than 1 and 2 don´t have a CIS instance in them. When using more than 2 clusters there is no need for additional CIS instances. The additional clusters will be externally managed by CIS. \n Notice as well that the picture only reflects a single BIG-IP. When using a BIG-IP HA pair, each BIG-IP will have its own CIS instances. This means that a multi-cluster setup will use 4 CIS instances usually. In the CIS High Availability section we will see how this can be reduced to 2 instances. \n Only the clusters with CIS instances will have the manifests that define the L7 routes (NextGen Routes or VirtualServer CRDs). On the other hand, all clusters that host application´s PODs wil need to have the Service manifests, these will be used by CIS to discover the workloads. \n CIS multi-cluster support works with Services of type either ClusterIP or NodePort. This article prefers the use of ClusterIP type as described in the article F5 BIG-IP deployment with OpenShift - platform and networking options. When using ClusterIP mode it is important to realize that the addressing of each cluster´s POD network cannot overlap with any other cluster. This must be taken into account prior to build the OpenShift clusters given that it is not possible to change the POD network of an OpenShift cluster after built. \n Multi cluster load balancing modes \nIn order to adapt to all customer use cases, the CIS can operate in the following modes:\n \n Active-Active: services in all clusters are treated equally. \n Ratio: clusters receive a different share of traffic which can be changed on the fly. \n Active-Standby: meant to be used when applications in the clusters cannot be simulateneously Active. \n \n These are ellaborated next. \n Active-Active mode \n In this mode a single pool is created for each application regardless in which cluster it is. In other words, each application has its own pool and its members are all the endpoints of the applications' Kubernetes Service in all clusters. Note in the next figures how all the PODs in all clusters are always used regardless the CIS HA state. \n Ratio mode \n Likewise with Active-Active mode, in this mode all the PODs in all clusters are always used regardless the CIS HA state. But unlike Active-Active mode, in this mode a given application will have a separate pool for each cluster which is the key for this multi-cluster method. For a given application, the load balancing is a two step process: \n \n The admin set ratio configured in CIS' global ConfigMap is used to select the service (a pool) from all clusters (set of pools). \n The LTM LB algorithm specified for the application in the manifests is used to LB within the selected pool. \n \n In other words, multi-cluster Ratio load balancing prioritizes the administrative decision first, then a regular dynamic load balancing is applied, for example round-robin, least-connections or application response time. \n Active-Standby mode \n This mode requires to have two CIS instances for each BIG-IP (in other modes only one CIS instance per BIG-IP is mandatory). In Active-Standby mode the workload PODs (applications) from the cluster where CIS is Standby are not considered. This mode could be used in Disaster Recovery deployments or when only applications from one of the clusters can be Active in a given moment (single-master applications). \n Although this mode is expected to be used typically for 2 cluster scenarios, it does support handling more than than 2 clusters. In such a case, the workloads in the additional clusters are always considered Active workloads. This is useful when there are non single-master applications as well (always-active or multi-master). The single-master applications would be placed in the clusters where the CIS instances are deployed and the always-active applications would be placed in all other clusters. This is shown in the next picture. \n CIS High Availability details \n When using CIS in multi-cluster mode it is possible to run two CIS instances in HA mode instead of the typical single CIS instance used in single cluster setups. These two CIS instances use the same image but they have Primary and Secondary roles (specified with the --multi-cluster-mode=primary or =secondary parameter). Normally, the Primary CIS is the instance that writes the configuration in the BIG-IP. Only when the Primary CIS instance fails the Secondary will take over and will take the role of writting the configuration in the BIG-IP. In order to detect CIS failures, two mechanisms are provided: \n \n The /ready health check endpoint in CIS. This endpoint returns HTTP 200 OK code if CIS can access to BIG-IP´s AS3 API and access to local cluster´s Kubernetes API . \n The Secondary CIS with the primaryEndPoint parameter is used to monitor Primary CIS' /ready endpoint. \n \n In order to expose the /ready endpoint this article recommends the use a NodePort type Service. This allows to monitor the /ready endpoint regardless in which OpenShift node the CIS instances are running. One Service for each CIS instance is required. \n Note that the above mechanisms doesn´t take into account if the BIG-IPs are in Active or Standby mode. \n In the rest of this section it will be described an optional HA-Groups configuration in the BIG-IP. This configuration would handle the corner case when two CIS instances are unable to push the configuration to BIG-IP \n The next pictures outline the resulting configuration with HA Groups when using redundant CIS instances for each BIG-IP: \n In this setup when the Primary CIS fails, the secondary CIS will take over and will be in charge of the configuration changes of the BIG-IP. The CIS instances of the peer BIG-IP work independently and continuously update its BIG-IP regardless it is Active or Standby. If the two CIS instances of the Active BIG-IP fail, then the BIG-IP HA Group setup will fail over the peer BIG-IP, if it has a CIS instance available. This is achieved with a simple HA Group configuration in the BIG-IP as follows, where each BIG-IP monitors its own CIS instances in a single pool and the values indicated in the sample HA Group configuration table of the previous picture. \n Notice from the screenshot above that it is showing the condition where one of its CIS instances is down, reducing the HA score to 10 yet this doesn´t produce a failover event thanks to the Active Bonus. More information on the BIG-IP HA-Group feature can be found here. \n The flexibility of BIG-IP and CIS HA mechanisms allow alternative configurations like monitoring a custom POD or having an external health endpoint to monitor the availability of a whole data centre. Another consideration to be taken into account is that when deploying CIS across different Availability Zones in a public cloud the F5 BIG-IP Cloud Failover should be used. \n 2-tier deployments \n 2-tier deployents were introduced in the article F5 BIG-IP deployment with OpenShift - platform and networking options. In these type of deployments, the BIG-IP sends the traffic to an ingress controller inside the cluster. This could be OpenShift´s Router (HA Proxy), NGINX+, Istio, or any other ingress like an API manager or a combination of these. BIG-IP can send the traffic to the appropiate ingress controller on a per-HTTP request basis using the oneconnect profile. To effectively accomplish this and monitor each service individually it is necessary to define the same L7 routes twice: in the tier-2 (the ingress controllers) and in the tier-1 (the BIG-IP). This is outlined in the next figure: \n To acomplish the above, the L7 routes in the BIG-IP are defined using the F5 VirtualServer CRD (as example) and it is also defined a separate Service for each L7 route, even if the ingress controller is the same for all L7 routes. This separate Service for each L7 route defines a pool in the BIG-IP for each Service and ultimately its own monitoring. \n Using the CIS multi-cluster feature and next steps \n To continue your journey with this unique feature, please check the official CIS documentation on this topic and official examples or the start-to-finish examples created by me in this GitHub repository. \n Conclusion and closing remarks \n As more applications are moved into OpenShift, more clusters will be needed in the enterprises and a flexible model to deploy the applications is required in which it is transparent the cluster in which a given application is being hosted. The CIS multi-cluster feature is a unique in the market and allows either complex scenarios such as clusters in stretched data centres or simplified OpenShift migrations with no application down time where upgrade in place is no longer required. Overall this feature is going to stay and there is no better time than now to be ready. \n \n \n ","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})@stringLength":"11972","kudosSumWeight":0,"repliesCount":0,"readOnly":false,"images":{"__typename":"AssociatedImageConnection","edges":[{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDE","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY4OTJpMkE5OTM0QkM5NDU5QTIwQg?revision=15\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDI","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MzRpMEJENTc2RTZFRTY3Qjc0Mg?revision=15\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDM","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MzVpQ0YyMzc2NkYxMkRDOUQzNA?revision=15\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDQ","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MjJpMzQ2QzNDOTc0NUMzODhEOQ?revision=15\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDU","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MjNpMDk0ODMyNTczMEQ3RDMwNw?revision=15\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDY","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MjVpNzA1MkFDODgzRDNCQjc3NA?revision=15\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDc","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MzFpRUM5NzdBN0U3MDIyQkFCNA?revision=15\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDg","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MThpMzNBNDU1REI3MDc0QzM2MA?revision=15\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDk","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MTlpNkU3RTBEMzk3MzEwNzhDRA?revision=15\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDEw","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5MjBpOUExQzEyQTZEQzI4Q0M1MA?revision=15\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDEx","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5NDNpNzg4NzExOTg3NUIxM0M5Qw?revision=15\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDEy","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5NDRpQkVGRDNDQkZERkZDNTgxNQ?revision=15\"}"}},{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDEz","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjU3NzQtMjY5NDZpNEMyRTJCMDM0Q0VFMzQyQQ?revision=15\"}"}}],"totalCount":13,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}},"videos":{"__typename":"VideoConnection","edges":[{"__typename":"VideoEdge","cursor":"MHxodHRwczovL3lvdXR1LmJlL05Oc1VmcUhnSkFNP3NpPWhEZGxOVXNXYm0wbEEtcUd8MHwyNTsyNXx8","node":{"__typename":"AssociatedVideo","videoTag":{"__typename":"VideoTag","vid":"https://youtu.be/NNsUfqHgJAM?si=hDdlNUsWbm0lA-qG","thumbnail":"https://i.ytimg.com/vi/NNsUfqHgJAM/hqdefault.jpg","uploading":false,"height":240,"width":320,"title":null},"videoAssociationType":"INLINE_BODY"}}],"totalCount":1,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}}},"Conversation:conversation:326367":{"__typename":"Conversation","id":"conversation:326367","topic":{"__typename":"TkbTopicMessage","uid":326367},"lastPostingActivityTime":"2024-01-17T11:34:07.242-08:00","solved":false},"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjYzNjctMjcxNTBpNEMyQ0EyNjg0OTNFODlDNQ?revision=9\"}":{"__typename":"AssociatedImage","url":"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjYzNjctMjcxNTBpNEMyQ0EyNjg0OTNFODlDNQ?revision=9","title":"CIS-multi-cluster.png","associationType":"BODY","width":2433,"height":1727,"altText":null},"TkbTopicMessage:message:326367":{"__typename":"TkbTopicMessage","subject":"Scale Multi-Cluster OpenShift Deployments with F5 Container Ingress Services","conversation":{"__ref":"Conversation:conversation:326367"},"id":"message:326367","revisionNum":9,"uid":326367,"depth":0,"board":{"__ref":"Tkb:board:TechnicalArticles"},"author":{"__ref":"User:user:326558"},"teaser@stripHtml({\"removeProcessingText\":true,\"truncateLength\":-1})":"","introduction":"","metrics":{"__typename":"MessageMetrics","views":290},"postTime":"2024-01-17T11:34:07.242-08:00","lastPublishTime":"2024-01-17T11:34:07.242-08:00","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})":" \n In the realm of enterprise IT, the complexity of managing applications across multiple clusters has long been a significant challenge. Recognizing this, we have introduced a solution with the latest release of BIG-IP Container Ingress Service (CIS). This article explores the challenges of distributing traffic across multiple clusters and how F5's CIS addresses these issues head-on. \n The Importance of Distributing Traffic Across Multiple Clusters \n Distributing traffic across multiple clusters is essential for maintaining high availability, ensuring consistent performance, and facilitating agile deployment strategies. However, achieving this is challenged by the complexity of configuring and managing multiple clusters, ensuring consistent application policies, and maintaining uptime during upgrades or deployments. \n Introducing the Solution: F5 BIG-IP CIS with Multi-Cluster Support \n F5's BIG-IP CIS now offers multi-cluster support, revolutionizing how applications are deployed across multiple OpenShift clusters. This new feature caters to a variety of deployment strategies and operational requirements: \n \n \n Deployment Strategies: Supports both Blue-Green Deployments and A/B Testing, allowing for more flexible and risk-averse application rollouts. \n \n \n High Availability (HA): Ensures continuous application availability by spanning across 'N' clusters, reducing the risk of downtime. \n \n \n Load Balancing: Implements Ratio-Based Load Distribution, optimizing resource utilization and ensuring efficient traffic management. \n \n \n Cluster Management: Facilitates Seamless Cluster Upgrades, ensuring that the latest features and security updates are implemented without impacting ongoing operations. \n \n \n How F5 BIG-IP CIS Works \n The picture below shows how the Multi-Cluster feature works. Notice that in the picture, we are showing 2 OpenShift Cluster, each with a CIS instance running. In a real deployment, we can run more than 2 clusters, but with only 2 CIS instances running in 2 of the clusters for redundancy purposes. \n Here is a brief workflow of the solution: \n \n \n Fetch KubeConfigs from ConfigMap: This step involves accessing the ConfigMap repository and extracting the necessary KubeConfig files for each external cluster. \n \n \n Subscribes to Informer Events: Utilizes the previously fetched KubeConfig to set up informers for external clusters. The informers continuously listen for and capture any events or changes occurring in these clusters, thereby maintaining a real-time monitoring system. \n \n \n Heartbeat Check for Primary CIS: The workflow includes regular heartbeat checks on the primary CIS. This involves sending periodic signals to the primary CIS to verify its operational status and ensure it is actively functioning. \n \n \n Post AS3 Declaration: In this final step, the system coordinates the posting of AS3 declarations to the external clusters. This is managed to ensure that only one CIS posts an AS3 declaration at any given time, thereby avoiding conflicts and ensuring orderly updates. \n \n \n Conclusion \n F5's BIG-IP CIS with Multi-Cluster Support is a strategic enabler for businesses relying on OpenShift. By simplifying the management of multi-cluster environments, organizations can focus more on innovation and less on multi-cluster application operational complexities. Watch our detailed demo on the F5 DevCentral YouTube channel to see how this solution can transform your OpenShift deployments. \n Explore the Demo: \n ","body@stripHtml({\"removeProcessingText\":true,\"removeSpoilerMarkup\":true,\"removeTocMarkup\":true,\"truncateLength\":-1})@stringLength":"3527","kudosSumWeight":2,"repliesCount":0,"readOnly":false,"images":{"__typename":"AssociatedImageConnection","edges":[{"__typename":"AssociatedImageEdge","cursor":"MjUuMnwyLjF8b3wyNXxfTlZffDE","node":{"__ref":"AssociatedImage:{\"url\":\"https://community.f5.com/t5/s/zihoc95639/images/bS0zMjYzNjctMjcxNTBpNEMyQ0EyNjg0OTNFODlDNQ?revision=9\"}"}}],"totalCount":1,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}},"videos":{"__typename":"VideoConnection","edges":[{"__typename":"VideoEdge","cursor":"MHxodHRwczovL3lvdXR1LmJlL3ZyWUFELWhNMHE4fDB8MjU7MjV8fA","node":{"__typename":"AssociatedVideo","videoTag":{"__typename":"VideoTag","vid":"https://youtu.be/vrYAD-hM0q8","thumbnail":"https://i.ytimg.com/vi/vrYAD-hM0q8/hqdefault.jpg","uploading":false,"height":113,"width":200,"title":null},"videoAssociationType":"INLINE_BODY"}}],"totalCount":1,"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}}},"CachedAsset:text:en_US-components/community/Navbar-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/community/Navbar-1743097588266","value":{"community":"Community Home","inbox":"Inbox","manageContent":"Manage Content","tos":"Terms of Service","forgotPassword":"Forgot Password","themeEditor":"Theme Editor","edit":"Edit Navigation Bar","skipContent":"Skip to content","migrated-link-9":"Groups","migrated-link-7":"Technical Articles","migrated-link-8":"DevCentral News","migrated-link-1":"Technical Forum","migrated-link-10":"Community Groups","migrated-link-2":"Water Cooler","migrated-link-11":"F5 Groups","Common-external-link":"How Do I...?","migrated-link-0":"Forums","article-series":"Article Series","migrated-link-5":"Community Articles","migrated-link-6":"Articles","security-insights":"Security Insights","migrated-link-3":"CrowdSRC","migrated-link-4":"CodeShare","migrated-link-12":"Events","migrated-link-13":"Suggestions"},"localOverride":false},"CachedAsset:text:en_US-components/community/NavbarHamburgerDropdown-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/community/NavbarHamburgerDropdown-1743097588266","value":{"hamburgerLabel":"Side Menu"},"localOverride":false},"CachedAsset:text:en_US-components/community/BrandLogo-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/community/BrandLogo-1743097588266","value":{"logoAlt":"Khoros","themeLogoAlt":"Brand Logo"},"localOverride":false},"CachedAsset:text:en_US-components/community/NavbarTextLinks-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/community/NavbarTextLinks-1743097588266","value":{"more":"More"},"localOverride":false},"CachedAsset:text:en_US-components/authentication/AuthenticationLink-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/authentication/AuthenticationLink-1743097588266","value":{"title.login":"Sign In","title.registration":"Register","title.forgotPassword":"Forgot Password","title.multiAuthLogin":"Sign In"},"localOverride":false},"CachedAsset:text:en_US-components/nodes/NodeLink-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/nodes/NodeLink-1743097588266","value":{"place":"Place {name}"},"localOverride":false},"CachedAsset:text:en_US-components/tags/TagSubscriptionAction-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/tags/TagSubscriptionAction-1743097588266","value":{"success.follow.title":"Following Tag","success.unfollow.title":"Unfollowed Tag","success.follow.message.followAcrossCommunity":"You will be notified when this tag is used anywhere across the community","success.unfollowtag.message":"You will no longer be notified when this tag is used anywhere in this place","success.unfollowtagAcrossCommunity.message":"You will no longer be notified when this tag is used anywhere across the community","unexpected.error.title":"Error - Action Failed","unexpected.error.message":"An unidentified problem occurred during the action you took. Please try again later.","buttonTitle":"{isSubscribed, select, true {Unfollow} false {Follow} other{}}","unfollow":"Unfollow"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageListTabs-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageListTabs-1743097588266","value":{"mostKudoed":"{value, select, IDEA {Most Votes} other {Most Likes}}","mostReplies":"Most Replies","mostViewed":"Most Viewed","newest":"{value, select, IDEA {Newest Ideas} OCCASION {Newest Events} other {Newest Topics}}","newestOccasions":"Newest Events","mostRecent":"Most Recent","noReplies":"No Replies Yet","noSolutions":"No Solutions Yet","solutions":"Solutions","mostRecentUserContent":"Most Recent","trending":"Trending","draft":"Drafts","spam":"Spam","abuse":"Abuse","moderation":"Moderation","tags":"Tags","PAST":"Past","UPCOMING":"Upcoming","sortBymostRecent":"Sort By Most Recent","sortBymostRecentUserContent":"Sort By Most Recent","sortBymostKudoed":"Sort By Most Likes","sortBymostReplies":"Sort By Most Replies","sortBymostViewed":"Sort By Most Viewed","sortBynewest":"Sort By Newest Topics","sortBynewestOccasions":"Sort By Newest Events","otherTabs":" Messages list in the {tab} for {conversationStyle}","guides":"Guides","archives":"Archives"},"localOverride":false},"CachedAsset:text:en_US-shared/client/components/common/QueryHandler-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-shared/client/components/common/QueryHandler-1743097588266","value":{"title":"Query Handler"},"localOverride":false},"CachedAsset:text:en_US-components/community/NavbarDropdownToggle-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/community/NavbarDropdownToggle-1743097588266","value":{"ariaLabelClosed":"Press the down arrow to open the menu"},"localOverride":false},"CachedAsset:text:en_US-shared/client/components/common/OverflowNav-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-shared/client/components/common/OverflowNav-1743097588266","value":{"toggleText":"More"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageView/MessageViewInline-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageView/MessageViewInline-1743097588266","value":{"bylineAuthor":"{bylineAuthor}","bylineBoard":"{bylineBoard}","anonymous":"Anonymous","place":"Place {bylineBoard}","gotoParent":"Go to parent {name}"},"localOverride":false},"CachedAsset:text:en_US-components/customComponent/CustomComponent-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/customComponent/CustomComponent-1743097588266","value":{"errorMessage":"Error rendering component id: {customComponentId}","bannerTitle":"Video provider requires cookies to play the video. Accept to continue or {url} it directly on the provider's site.","buttonTitle":"Accept","urlText":"watch"},"localOverride":false},"CachedAsset:text:en_US-components/users/UserLink-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/users/UserLink-1743097588266","value":{"authorName":"View Profile: {author}","anonymous":"Anonymous"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageSubject-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageSubject-1743097588266","value":{"noSubject":"(no subject)"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageTime-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageTime-1743097588266","value":{"postTime":"Published: {time}","lastPublishTime":"Last Update: {time}","conversation.lastPostingActivityTime":"Last posting activity time: {time}","conversation.lastPostTime":"Last post time: {time}","moderationData.rejectTime":"Rejected time: {time}"},"localOverride":false},"CachedAsset:text:en_US-shared/client/components/nodes/NodeIcon-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-shared/client/components/nodes/NodeIcon-1743097588266","value":{"contentType":"Content Type {style, select, FORUM {Forum} BLOG {Blog} TKB {Knowledge Base} IDEA {Ideas} OCCASION {Events} other {}} icon"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageUnreadCount-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageUnreadCount-1743097588266","value":{"unread":"{count} unread","comments":"{count, plural, one { unread comment} other{ unread comments}}"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageViewCount-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageViewCount-1743097588266","value":{"textTitle":"{count, plural,one {View} other{Views}}","views":"{count, plural, one{View} other{Views}}"},"localOverride":false},"CachedAsset:text:en_US-components/kudos/KudosCount-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/kudos/KudosCount-1743097588266","value":{"textTitle":"{count, plural,one {{messageType, select, IDEA{Vote} other{Like}}} other{{messageType, select, IDEA{Votes} other{Likes}}}}","likes":"{count, plural, one{like} other{likes}}"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageRepliesCount-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageRepliesCount-1743097588266","value":{"textTitle":"{count, plural,one {{conversationStyle, select, IDEA{Comment} OCCASION{Comment} other{Reply}}} other{{conversationStyle, select, IDEA{Comments} OCCASION{Comments} other{Replies}}}}","comments":"{count, plural, one{Comment} other{Comments}}"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageBody-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageBody-1743097588266","value":{"showMessageBody":"Show More","mentionsErrorTitle":"{mentionsType, select, board {Board} user {User} message {Message} other {}} No Longer Available","mentionsErrorMessage":"The {mentionsType} you are trying to view has been removed from the community.","videoProcessing":"Video is being processed. Please try again in a few minutes.","bannerTitle":"Video provider requires cookies to play the video. Accept to continue or {url} it directly on the provider's site.","buttonTitle":"Accept","urlText":"watch"},"localOverride":false},"CachedAsset:text:en_US-shared/client/components/users/UserAvatar-1743097588266":{"__typename":"CachedAsset","id":"text:en_US-shared/client/components/users/UserAvatar-1743097588266","value":{"altText":"{login}'s avatar","altTextGeneric":"User's avatar"},"localOverride":false}}}},"page":"/tags/TagPage/TagPage","query":{"nodeId":"board:TechnicalArticles","tagName":"Red Hat OpenShift"},"buildId":"q_bLpq2mflH0BeZigxpj6","runtimeConfig":{"buildInformationVisible":false,"logLevelApp":"info","logLevelMetrics":"info","openTelemetryClientEnabled":false,"openTelemetryConfigName":"f5","openTelemetryServiceVersion":"25.2.0","openTelemetryUniverse":"prod","openTelemetryCollector":"http://localhost:4318","openTelemetryRouteChangeAllowedTime":"5000","apolloDevToolsEnabled":false,"inboxMuteWipFeatureEnabled":false},"isFallback":false,"isExperimentalCompile":false,"dynamicIds":["./components/customComponent/CustomComponent/CustomComponent.tsx","./components/community/Navbar/NavbarWidget.tsx","./components/community/Breadcrumb/BreadcrumbWidget.tsx","./components/tags/TagsHeaderWidget/TagsHeaderWidget.tsx","./components/messages/MessageListForNodeByRecentActivityWidget/MessageListForNodeByRecentActivityWidget.tsx","./components/tags/TagSubscriptionAction/TagSubscriptionAction.tsx","./components/customComponent/CustomComponentContent/TemplateContent.tsx","../shared/client/components/common/List/ListGroup/ListGroup.tsx","./components/messages/MessageView/MessageView.tsx","./components/messages/MessageView/MessageViewInline/MessageViewInline.tsx","./components/customComponent/CustomComponentContent/HtmlContent.tsx","./components/customComponent/CustomComponentContent/CustomComponentScripts.tsx"],"appGip":true,"scriptLoader":[]}