F5 Distributed Cloud and Amazon FSx for NetApp ONTAP - Global Replication and Anywhere Access
F5 Distributed Cloud (XC) is a SaaS solution which securely exposes services to the correct service consumers, whether the endpoints of the communications are in public clouds, private clouds, or on-prem data centers. In many cases, distributed load balancers are tactical XC tools to proxy published services, like a web application on TCP port 443 to backend, secured origin pools. In other cases, a nimble layer 3 service, along the lines of a next-generation multi-site turnkey VPN solution are equally attractive, where protocols like SnapMirror might consume a small range of TCP ports to empower backup and replication jobs between enterprise endpoints. This article explores how to easily set up a replication solution for islands of NetApp storage, such as ONTAP clusters located at enterprise offices and data centers. The end state is all critical data securely replicated to the cloud-native Amazon FSx for NetApp ONTAP offering. Beyond backup and recovery purposes, the solution may in turn allow local AWS-instantiated read/write access to key global volumes with the highest possible QoS guarantees. Such a centralized corralling of critical data might align with feeding corporate AI RAG initiatives that infuses LLMs with private corporate data sources to reach optimal AI context-specific AI outcomes. Technology Overview – Leverage Secure Layer Three Multicloud Connections The XC Network Connect module harnesses the built-out global F5 infrastructure to provide automatic reachability between customer edge (CE) sites on-prem, in private cloud or enterprise tenants within an array of public cloud providers. Where WAN facilities may already exist, CE sites can also be directly inter-connected with those data planes. A key deliverable is that routing is automatic and those simply troubleshooting and analytic tools required by NetOps are uniform and available as a single-pane-of-glass experience in the XC SaaS console. Network segmentation is essential in controlling modern and complex enterprise environments. Traditional methods like Virtual Routing and Forwarding (VRFs) and Multiprotocol Label Switching (MPLS) have long been used to create isolated network segments in on-prem setups. F5 XC ensures segmentation in environments like AWS, and it can extend the same segmentation to on-prem environments. These techniques separate traffic, enhance security, and improve network management by preventing unauthorized access and minimizing the attack surface. Any interface on a CE node, such as in an office computing room, may be associated with an XC “segment”, essentially a VRF setup. For instance, segments might be labeled production, development, and quality assurance. Distributed Cloud now supports up to eight Ethernet interfaces on physical or virtual appliances, along with VLAN-tagged sub-interfaces. As such, the number of segments required is not normally a concern. Azure VNETs may also be included in segments. With AWS, transit gateway (TGW) sites are particularly useful for high scalability, as they allow a hub-and-spoke architecture for routing decisions. As such, just one single CE site in a TGW virtual private cloud (VPC) can enable reachability to countless segments. The following is a simple example of four segments simply labeled as color names, including a CE site in the lower right with attachments to multiple segments. It is common to require interconnectivity between segments, in other words, to “leak” routes automatically between selected segments in certain scenarios. A use case might be a shared VPC where modern logging receivers and analytics solutions reside. In the following simple case, it can be seen that both production and development segments, which span AWS and on-prem CE sites, may need to interact with the shared VPC. The mentioned common reachability requirement, allowing shared services such as logging to interact with other segments, does not require any routing knowledge or command-line tools. A simple “segment connector” GUI within the XC console allows simplified workflows. In the following XC screenshot, two existing segments are selected for mutual reachability and options are direct or to perform a SNAT operation to adjust source IP addresses. Replicating Global NetApp Volumes to Amazon FSx for NetApp ONTAP Amazon FSx for NetApp ONTAP, or simply FSx for ONTAP for brevity, allows a range of benefits to enterprise, including true multiprotocol support for accessing data, including NFS, SMB, S3, iSCSI, and NVMe-over-TCP protocols. This AWS-based NetApp service offering also benefits from scalability control. One may support their growing data sets with storage capacity that grows and shrinks automatically, no day-to-day manual intervention required. One of the principal use cases is controlled migration of apps to cloud environments; popular choices in moving some applications to AWS include highly scalable web applications, data-intensive applications requiring large storage and processing power, and real-time applications with high traffic; applications with complex data analytics needs. One of the differentiating features of FSx for ONTAP is that beyond being a scalable cloud-hosted storage solution, it integrates very tightly with long-established on-prem NetApp solutions, key NetApp workflows like: Back up, archive, or replicate data from your on-premises file servers to FSx for ONTAP, harnessing mature SnapMirror technology to simplify business continuity and meet modern data retention and disaster recovery requirements. Data in an on-premises NetApp file system that requires access or processing from AWS hosts or services with ultra-low latency, configure FSx for ONTAP as an in-cloud cache for your valuable on-premises data by using NetApp FlexCache. When used as a cache, FSx for ONTAP provides lightning-fast access to your popular on-premises data sets from AWS compute instances. Embrace cloud cost efficiencies: Unlike generally fixed costs with growing on-prem deployments that may be overprovisioned, pay only for the resources you actually use, with various pricing models to optimize costs. F5 Distributed Cloud will serve as the underlay, turn-key networking solution to bind NetApp elements, virtually anywhere around the globe, with FSx for ONTAP. The following is a simple deployment shown visually, with the intention of SnapMirroring key enterprise volumes around the world to FSx for ONTAP instantiated on AWS East-2. Replicate AI Training Data from On-Prem to AWS FSx and Locally Utilize with EC2 Compute A key rationale for replicating data from on-prem to AWS is disaster recovery. A typical architecture selected for FSx for ONTAP is to spread highly-available volumes across multiple availability zones (AZ) for the utmost in redundancy. In this example, a volume representing files used in a RAG AI document volume is replicated to AWS. This end goal can be achieved in many approaches, including NetApp BlueXP console, or using the local AWS console and on-prem NetApp System Manager to create the replication. In this particular case, a simple command-line approach is utilized and can be found documented here. The source NetApp ONTAP Select is located in the Seattle area and a requirement exists to Snap Mirror volumes to FSx for ONTAP. With a pair of F5 CE nodes, the red dashed line above indicates the required L3 reachability. With CE sites deployed, the configuration of the production segment is done on the following screen in the Multicloud Network Connect module of the XC console. In the above, one notes the production segment consists in total of two VPCs and two physical locations, Seattle (Redmond, WA) and San Jose. The “Allow traffic from this segment to Internet” indicates hosts on the attached subnets will be permitted outbound Internet access, in much the same way a NAT Gateway can optionally facilitate this in many common AWS deployments. Looking at the NetApp traditional System Manager Volumes view, we isolate the volume which we wish to replicate to FSx for ONTAP, RAG_Secure_Documents. Through the FSx for ONTAP command interface, we start by creating a volume in the AWS offering to receive replicated content. This is a data protection volume as denoted by type “DP”: FSx-Dest::> vol create -vserver fsx -volume RAG_Secure_Documents -aggregate aggr1 -size 1g -type DP -tiering-policy all We then establish cluster peering between the on-prem ONTAP Select and the FSx for ONTAP offering, through similar commands on each end: FSx-Dest::> cluster peer create -address-family ipv4 -peer-addrs 10.50.0.221 Seattle ONTAP::> cluster peer create -address-family ipv4 -peer-addrs 10.1.200.36,10.1.200.202 After peering, the storage virtual machines (SVMs) that govern the volumes, both on-prem and in AWS, we can then establish the SnapMirror relationship and transfer the data to replicate our volume, via F5 XC, between Seattle and FSx for ONTAP. FSx-Dest::>snapmirror create -source-path svm0:RAG_Secure_Documents -destination-path fsx:RAG_Secure_Documents -vserver fsx -throttle unlimited FSx-Dest::>snapmirror initialize -destination-path fsx:RAG_Secure_Documents -source-path svm0:RAG_Secure_Documents The following command, applied against FSx for ONTAP, demonstrates a successful and on-going SnapMirror relationship from on-prem to AWS for the volume “RAG_Source_Documents_2024”: FsxId05954ec2335664e17::> snapmirror show -fields state,status source-path destination-path state status ------------------------- ------------------------ ---------- ----- svm0:RAG_Secure_Documents fsx:RAG_Secure_Documents Broken-off Idle svm0:RAG_Source_Documents_2024 fsx:RAG_Source_Documents_2024 Snapmirrored Idle The fact that the other volume, RAG_Secure_Documents has been broken off stems from the fact that this volume in FSx for ONTAP has been flipped from a data protection (DP) volume to an actively used read-write (RW) volume. Perhaps EC2 instances that are heavily steeped in AI workloads can benefit from AWS-housed volumes and a single-point of reference for global islands of NetApp volumes centralized to FSx for ONTAP for fixed-length projects. In this case, a standard NFS mount command will allow any EC2 host to consume the data: The above are just two examples of the multi-faceted value of corralling enterprise NetApp data volumes with F5, SnapMirror for the purposes of replication and disaster preparedness, coupled with the value of centralized, read-write copies of valuable data. FlexCache – High Performance Use of Worldwide NetApp Volumes centrally Accessed via F5 and FSx for ONTAP Caching is a valuable longstanding tool to make content that might otherwise travel across WANs, including trans-continental links, appear local, with staleness checks occurring automatically to validate the latest files are always used. One of the many scenarios potentially leveraging NetApp’s FlexCache technology is explored in this section. Consider workflows that are housed in AWS East-2, based in Ohio that make frequent use of files physically housed on a west coast ONTAP cluster in the Seattle area, in Redmond, Washington. F5 Distributed Cloud allows for seamless interconnections of enterprise data volumes, however, there is much to be gained by AWS East-2 EC2 hosts consuming local versions of the west coast content whenever possible. The laws of physics impose constraints that even modern fiber-optic networks must fall within. The distance between Seattle and Columbus, Ohio is approximately 3,200 km (almost 2,000 miles) and in 2025 on-going measurements of ICMP response times put latency in the 60 millisecond region (link). Trans-oceanic latency will be even more substantial and NAS access protocols like Server Message Block (SMB) are notoriously chatty, with end-to-end application message pairs required to enable even simple, single-file copy jobs. All popular NAS protocols, regardless of application versions, ride upon TCP, which itself has an ongoing round-trip acknowledgment tax, which imposes a higher penalty for long WAN connections. With FlexCache volumes configured in a central location, where end user consumption is highest, in our example AWS East-2, the first read of files from the west coast will be bound by the above laws of physics. However, subsequent reads by users of the same files will be served out of the FlexCache volume in FSx, thereby making load or copies times negligible for “hot” content. To demonstrate this, 25 megabytes of Gutenberg Project free-access novels were uploaded via NFS to the Redmond ONTAP Select appliance via a local client. The following FlexCache volume was created on FSx for ONTAP, following the published guidelines for AWS FSx found here. (at FSx command line): vserver peer create -vserver svmfsx0 -peer-server svm2 -peer-cluster f5netappclusterE -local-name svm2 -application flexcache (at west coast ONTAP Select command line):vserver peer accept -vserver svm2 -peer-vserver svmfsx0 -local-name svmfsx0 volume flexcache create -vserver svmfsx0 -size 15g -volume CacheVol -origin-volume seattleofficefiles -origin-vserver svm2 -junction-path /flexcache -aggr-list aggr1 The above screenshot demonstrates the result of the commands, a new volume “CacheVol” associated with one of the FSx for ONTAP storage virtual machines and accessible to EC2 or other hosts at the path “/flexcache”. The NetApp FlexCache inner workings, including discussions around the RAL protocol, which underpins cache management, are discussed here. A test EC2 host in AWS East-2 has mounted the FlexCache volume “CacheVol” using the following standard command sequence. # sudo mkdir -p /seattle-files #sudo mount -t nfs -o nfsvers=3 10.1.200.104:/flexcache /seattle-files The 25-megabytes of books are copied to a local folder on the EC2 with the following command. # cd /seattle-files # cp *.epub /local-file-copies The resulting network activity, copying the remote files to the EC2 while simultaneously automatic loading of the CacheVol is seen through the F5 XC Flow Analyzer, which has been set to report on a 10-minute window of traffic leaving the Seattle CE site (s-gorman-redmond-smsv2-ce) and targeting the AWS East-2 Transit Gateway (TGW) CE site (s-gorman-tgw-may25). One can see the data totals reflect the large file transfer, being close to 25 megabytes, however the TCP connections as expected do not use the NAS access protocol port, typically TCP port 2049. Instead, TCP 11105, the well-known port used traditionally by SnapMirror, is leveraged. This reminds one that block-level optimizations like data compression will be enacted. Multiple connections are made use of, client-side ephemeral ports as seen in this case are in the 20444 and up range. A few minutes later, the EC2 host was instructed again to copy these publications to another, different local folder. In this case, the copy process was virtually instantaneous from the user’s perspective. The 25 megabytes of file were immediately seen in the local folder. Analyzing the F5 Flow Analysis tool for the same amount of time, ten minutes, covering the span of this second copy, we see negligible traffic crossing the continent; caching has proved its value. The F5 flow analyzer is a nice, simple asset to other integrated tools such as traceroute or TCPDump; for instance the flow analyzer can report on individual IP addresses with top talkers or top listeners isolated over 5 minutes, 1 hour, or 24 hours all available. Summary In this article, we have demonstrated that the F5 Distributed Cloud Network Connect module, a modern approach to layer 3 multicloud networking (MCN) can be the turnkey glue for global NetApp storage elements, a simple networking end-state that is both performant and heightens security through segmentation. In alignment with this is the simple cloud-backed NetApp storage solution Amazon FSx for NetApp ONTAP, that uses well-known NetApp workflows to setup both NAS (NFS, SMB) and SAN (iSCSI) access. It concurrently offers a consolidated suite of storage management tools such as disaster recovery for islands of traditional ONTAP appliances throughout an enterprise’s global footprint. Three principal use cases were demonstrated in this article. First, a disaster recovery approach through on-going SnapMirror sessions from remote sites to a centralized AWS FSx cloud-native highly available solution. Second, the data protection volumes created through SnapMirror were easily re-purposed to offer classic read-write volumes accessible through standard NAS protocols. A final use case was the ability to enable NetApp FlexCache volumes to provide lightning-quick access to distant volumes through F5 XC MCN. As data from offices, such as a west coast Seattle site, traveled towards consumption, for example east coast EC2 instances within AWS, FlexCache quickly populated the FlexCache volumes within FSx for ONTAP. Any subsequent access requests to this West Coast data were observed to be provided through local, ultra-low-latency access times typical of intra-AWS region communications. F5 has provided a set of console views in the XC SaaS interface to monitor the underlying NetApp flows as they pass through the Distributed Cloud infrastructure.Deploying F5 Distributed Cloud Customer Edge in Red Hat OpenShift Virtualization
Introduction Red Hat OpenShift Virtualization is a feature that brings virtual machine (VM) workloads into the Kubernetes platform, allowing them to run alongside containerized applications in a seamless, unified environment. Built on the open-source KubeVirt project, OpenShift Virtualization enables organizations to manage VMs using the same tools and workflows they use for containers. Why OpenShift Virtualization? Organizations today face critical needs such as: Rapid Migration: "I want to migrate ASAP" from traditional virtualization platforms to more modern solutions. Infrastructure Modernization: Transitioning legacy VM environments to leverage the benefits of hybrid and cloud-native architectures. Unified Management: Running VMs alongside containerized applications to simplify operations and enhance resource utilization. OpenShift Virtualization addresses these challenges by consolidating legacy and cloud-native workloads onto a single platform. This consolidation simplifies management, enhances operational efficiency, and facilitates infrastructure modernization without disrupting existing services. Integrating F5 Distributed Cloud Customer Edge (XC CE) into OpenShift Virtualization further enhances this environment by providing advanced networking and security capabilities. This combination offers several benefits: Multi-Tenancy: Deploy multiple CE VMs, each dedicated to a specific tenant, enabling isolation and customization for different teams or departments within a secure, multi-tenant environment. Load Balancing: Efficiently manage and distribute application traffic to optimize performance and resource utilization. Enhanced Security: Implement advanced threat protection at the edge to strengthen your security posture against emerging threats. Microservices Management: Seamlessly integrate and manage microservices, enhancing agility and scalability. This guide provides a step-by-step approach to deploying XC CE within OpenShift Virtualization, detailing the technical considerations and configurations required. Technical Overview Deploying XC CE within OpenShift Virtualization involves several key technical steps: Preparation Cluster Setup: Ensure an operational OpenShift cluster with OpenShift Virtualization installed. Access Rights: Confirm administrative permissions to configure compute and network settings. F5 XC Account: Obtain access to generate node tokens and download the XC CE images. Resource Optimization: Enable CPU Manager: Configure the CPU Manager to allocate CPU resources effectively. Configure Topology Manager: Set the policy to single-numa-node for optimal NUMA performance. Network Configuration: Open vSwitch (OVS) Bridges: Set up OVS bridges on worker nodes to handle networking for the virtual machines. NetworkAttachmentDefinitions (NADs): Use Multus CNI to define how virtual machines attach to multiple networks, supporting both external and internal connectivity. Image Preparation: Obtain XC CE Image: Download the XC CE image in qcow2 format suitable for KubeVirt. Generate Node Token: Create a one-time node token from the F5 Distributed Cloud Console for node registration. User Data Configuration: Prepare cloud-init user data with the node token and network settings to automate the VM initialization process. Deployment: Create DataVolumes: Import the XC CE image into the cluster using the Containerized Data Importer (CDI). Deploy VirtualMachine Resources: Apply manifests to deploy XC CE instances in OpenShift. Network Configuration Setting up the network involves creating Open vSwitch (OVS) bridges and defining NetworkAttachmentDefinitions (NADs) to enable multiple network interfaces for the virtual machines. Open vSwitch (OVS) Bridges Create a NodeNetworkConfigurationPolicy to define OVS bridges on all worker nodes: apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: ovs-vms spec: nodeSelector: node-role.kubernetes.io/worker: '' desiredState: interfaces: - name: ovs-vms type: ovs-bridge state: up bridge: allow-extra-patch-ports: true options: stp: true port: - name: eno1 ovn: bridge-mappings: - localnet: ce2-slo bridge: ovs-vms state: present Replace eno1 with the appropriate physical network interface on your nodes. This policy sets up an OVS bridge named ovs-vms connected to the physical interface. NetworkAttachmentDefinitions (NADs) Define NADs using Multus CNI to attach networks to the virtual machines. External Network (ce2-slo): External Network (ce2-slo): Connects VMs to the physical network with a specific VLAN ID. This setup allows the VMs to communicate with external systems, services, or networks, which is essential for applications that require access to resources outside the cluster or need to expose services to external users. apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: ce2-slo namespace: f5-ce spec: config: | { "cniVersion": "0.4.0", "name": "ce2-slo", "type": "ovn-k8s-cni-overlay", "topology": "localnet", "netAttachDefName": "f5-ce/ce2-slo", "mtu": 1500, "vlanID": 3052, "ipam": {} } Internal Network (ce2-sli): Internal Network (ce2-sli): Provides an isolated Layer 2 network for internal communication. By setting the topology to "layer2", this network operates as an internal overlay network that is not directly connected to the physical network infrastructure. The mtu is set to 1400 bytes to accommodate any overhead introduced by encapsulation protocols used in the internal network overlay. apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: ce2-sli namespace: f5-ce spec: config: | { "cniVersion": "0.4.0", "name": "ce2-sli", "type": "ovn-k8s-cni-overlay", "topology": "layer2", "netAttachDefName": "f5-ce/ce2-sli", "mtu": 1400, "ipam": {} } VirtualMachine Configuration Configuring the virtual machine involves preparing the image, creating cloud-init user data, and defining the VirtualMachine resource. Image Preparation Obtain XC CE Image: Download the qcow2 image from the F5 Distributed Cloud Console. Generate Node Token: Acquire a one-time node token for node registration. Cloud-Init User Data Create a user-data configuration containing the node token and network settings: #cloud-config write_files: - path: /etc/vpm/user_data content: | token: <your-node-token> slo_ip: <IP>/<prefix> slo_gateway: <Gateway IP> slo_dns: <DNS IP> owner: root permissions: '0644' Replace placeholders with actual network configurations. This file automates the VM's initial setup and registration. VirtualMachine Resource Definition Define the VirtualMachine resource, specifying CPU, memory, disks, network interfaces, and cloud-init configurations. Resources: Allocate sufficient CPU and memory. Disks: Reference the DataVolume containing the XC CE image. Interfaces: Attach NADs for network connectivity. Cloud-Init: Embed the user data for automatic configuration. Conclusion Deploying F5 Distributed Cloud CE in OpenShift Virtualization enables organizations to leverage advanced networking and security features within their existing Kubernetes infrastructure. This integration facilitates a more secure, efficient, and scalable environment for modern applications. For detailed deployment instructions and configuration examples, please refer to the attached PDF guide. Related Articles: BIG-IP VE in Red Hat OpenShift Virtualization VMware to Red Hat OpenShift Virtualization Migration OpenShift VirtualizationMigrating between sites using stretched VLANs
Background Recently I was asked the following question by a customer: I have 2 physical datacenters. Datacenter A is a legacy datacenter and almost all workloads run here, and Datacenter B is a new location with almost no workloads deployed. I must migrate quickly out of location A into B, in order to close location A. Using NSX-T, I have "stretched" VLAN's between sites, meaning that a single VLAN and CIDR block is available to VM's in either datacenter. I have 2x F5 BIG-IP devices configured in an HA pair (Active/Standby) in location A, but none yet in B. I have thousands of Virtual Servers configured on my BIG-IP devices, and many thousands of VM's configured as pool members, all running in location A. I can migrate workloads by replicating VM's across datacenters, leaving their IP addresses unchanged after migration to location B. What are my options for migrating the BIG-IP devices to location B? I'd like to maintain High Availability within each datacenter, minimize any traffic between the sites, and minimize operational difficulties. Let's take a look at these requirements and review potential solutions. Defining our migration Firstly, let's define our existing environment and challenges in a little more detail. What is a stretched VLAN? "Stretched VLAN" is a shorthand phrase for the practice of extending Layer 3 networks across physical sites. This is useful in situations like VM migration across data centers without needing to change the VM's network configuration. If datacenters are within a few miles of each other, direct Layer 2 connectivity may be possible. A more commonly preferred approach is tunneling Layer 2 networks across routed (Layer 3) networks. This allows for more control over networking and relieves some constraints of direct L2 connections. The primary technology used to stretch a VLAN across physical data centers is VxLAN, which allows for Layer 2 connectivity extension over a Layer 3 network, effectively creating a virtual overlay network that can span multiple data centers while maintaining logical segmentation within the VLANs. VxLAN is used by VMware's NSX-T offering, but other technologies also exist, such as L2TPv3 and MPLS. How can NSX-T minimize inter-site traffic? Because NSX-T can have logically distributed routers, we can define an overlay-backed segment. In an overlay-backed segment, traffic between two VMs on different hosts but attached to the same overlay segment has their layer 2 traffic carried by a tunnel between the hosts. directly, bypassing the physical VLAN's. Diagram source In practice, this means that traffic between two VM's in the same segment - even if they are on different VLANs and different ESXi hosts - does not need to traverse the physical network gateway. I.e, if our VLAN's default gateway of ".1" exists on a physical router in Location A, but traffic is being sent between 2x VM's on different hosts and VLAN's in Location B, the traffic does not need to traverse the inter-site link. This is very powerful. To minimize VM-to-VM traffic crossing the inter-site link, we must configure NSX-T correctly, migrate all VM's in a given app/workload between data centers at the same time, and also at the same time move any F5 VIPs that process application traffic for that workload. Pre-migration BIG-IP overview The simplified diagram below shows the customer's pre-migration environment. The legacy datacenter still hosts almost all workloads, but the plumbing has been laid for migration. Using NSX-T, the VLANs in datacenter A are stretched to datacenter B. Existing BIG-IP's can reach pool members in Datacenter B. Any VM can be migrated between sites A and B without changing it's IP address. Post-migration BIG-IP end state The diagram below shows our post-migration goal. Let's remember our requirements. We want to get here with: HA maintained within each datacenter (we'll have 4 devices for a period of time) Minimal inter-site traffic (skip the physical VLAN gateway if possible) The easiest method possible (ie, sync configs and do not deploy new config on a disparate BIG-IP cluster) Migration options Let's review a few options for migration. 1. HA pair across physical locations Given that our end goal is to have all VM workloads and BIG-IP VE's in location B, we may be tempted to take a shortcut approach: just migrate one BIG-IP VE to site B and run an Active/Standby pair across datacenters. This could work in terms of connectivity, but raises some disadvantages for a production scenario: Latency. Only 1 BIG-IP can be Active. Latency will occur for all traffic between the Active BIG-IP and any nodes in the other datacenter. HA. Running a single HA pair across sites leaves us without High Availability within either site. Hard cutover. A cutover of the Active/Standy roles between site A to site B can be planned, but it's an "all or nothing" approach in terms of site A or B hosting the Active BIG-IP. There's no graceful way to keep both VM workloads and the Active BIG-IP in the same site and migrate together. I have personally managed HA pairs run across two physical datacenters with Layer 2 connectivity. In a scenario with very little latency between sites, or where a migration was not planned, that might be appropriate. However, in this case, the disadvantages listed here make this option less than ideal. 2. Second HA pair of BIG-IP devices in site B Given that our end goal is to have a single HA pair of devices in site B, we could build a brand new, disparate BIG-IP cluster in site B and migrate Virtual Servers from cluster A to B. After migration, decommission cluster A. This could work but raises unnecessary complications: IP conflicts. If we build 2x HA pairs, both on the same VLAN's, we have the potential for IP conflicts. We must migrate every Virtual Server and Pool by recreating our configuration on cluster B. This means every VIP must change. Tediousness. We could alleviate some pain by cleverly replicating the configuration from cluster A on cluster B, but disabling Virtual Addresses until a time of cutover. This would be possible but tedious, and introduce some risk. We could automate the deletion of VIP's and pools from one device, and creation on another, but this automation is likely to be difficult if the original configuration was not created with automation itself. DNS. If IP addresses of VirtualServers do change, we must consider the DNS changes that would be required. One advantage, however, is that two separate clusters is an architecture that is very easy to understand. 3. Single device cluster with 4x BIG-IPs This is my preferred approach when latency between sites is non-trivial and we must operate for some time with Active BIG-IPs in both locations. We'll temporarily grow our cluster from 2 devices to 4, with 2 devices in each cluster. We'll introduce an additional Traffic Group. Traffic Group 1 (existing TG) will be Active on BIG-IP 1 with BIG-IP 2 as next preferred. Traffic Group 2 (new TG) will be Active on BIG-IP 3 with BIG-IP 4 as next preferred. Pre-migration, all VIP's exist in TG 1. Migrate workloads components together: Migrate related VM's for workloads to Datacenter B. Migrate appropriate VIP's between TG 1 and TG 2. East-West traffic between workload VMs and BIG-IP should all remain within Location B. Once all VIP's are migrated to Traffic Group 2: Delete Traffic Group 1 Remove and decommission BIG-IP's 1 and 2. The advantages to his approach are: No IP address changes. Virtual Addresses of Virtual Servers do not change. Operationally smooth. A TMSH command can move a Virtual Address between Traffic Groups: tmsh modify ltm virtual-address 192.0.2.1 traffic-group <desired_traffic_group> See K53040085: Workloads can move app-by-app and not in an "all-or-nothing" approach. No DNS changes required. HA maintained within sites. Further details when configuring a single, 4-Device Cluster The basic idea behind this migration architecture is that a given VIP will be advertised via ARP from either Location A or B, depending on the Traffic Group to which the Virtual Address belongs. This allows us to have a single Device Group (easiest because config is synced between all devices) and to use two Traffic Groups. The Device Group type is still Sync-Failover (and not Sync-Only). In a Sync-Only group, the /Common folder is not synced between member devices, but in our case the existing configuration exists within /Common and so we will need this replicated between all four devices. Multiple Device Groups within a single Trust Domain, as diagrammed in this example, are not planned in this scenario. Because partitions are mapped to a Device Group, and all of our existing configuration is within a single /Common partition, multiple Device Groups are not appropriate in this case. Failover is configured individually for each Traffic Group. TG 1 will failover between BIG-IP 1 & 2, and TG 2 will failover between BIG-IP 3 & 4. We often refer to two-node clusters as "Active/Standby" or "Active/Active". When a cluster has 3+ nodes we often refer to it as "ScaleN" or "Active/Active/Standby", or similar. For this scenario, we might use the term "Active/Standby/Active/Standby", since we'll have 2 Active devices and 4 devices total during our migration. Further questions for our customer scenario When do we migrate the default gateway for each of our physical VLANs? The physical VLAN's have a gateway - let's call it ".1" - currently configured on routers on Virginia. This is relevant because some traffic may traverse physical VLAN's: traffic between VM's that are actually running in different datacenters because they were not migrated together, traffic from regional sites that are routed by the WAN to Virginia, traffic to physical VM's that will be migrated separately, etc. Moving that ".1" from Virginia to Dallas will be once-off move that's best left to the customer's network team. Advanced network management technologies can distribute a VLAN and gateway for physical networks, just like our NSX-T example does for VLANs within the virtual environment. But this decision is outside of the recommendation for F5 BIG-IP migration. What about physical servers in Virginia? How do we handle those? Physical machines must be re-built in Location 2. Physical VLAN's can be stretched, just like VLAN's within the virtual environment. Or, physical machines may reside on VLAN's that are local to each datacenter. In the case of a stretched physical VLAN, a physical server that is a pool member in BIG-IP could maintain it's IP address and BIG-IP configuration would not change. If the IP address of a pool member does change, of course the BIG-IP configuration must be updated. This makes the migration of physical servers less automated than VM's. In this scenario, the BIG-IP's themselves are virtual machines. If they were physical appliances on a stretched VLAN, the same approach would apply for BIG-IP migration (single Device Groups with two Traffic Groups). What about self IP addresses? This is a potentially important detail. Each BIG-IP will have unique Self IP's in each VLAN to which it is connected. Let's walk through this from two perspectives: health monitoring and SNAT. Health monitoring of pool members will be sourced from the non-floating Self IP that is local to the Active BIG-IP. Health monitoring is not sourced from a floating Self IP, if one exists. By introducing two additional devices we will be adding two IP addresses from which pool members may receive health checks. In our customer's case, no firewall currently exists between BIG-IP and pool members, so it's very unlikely that health monitoring from a new IP address will break an application. But it is important to note: if an application is expecting health monitoring to come from one of the two existing BIG-IP Self IP's only, we may need to update that application. SNAT, which is extremely common, means connections between BIG-IP and pool members will be sourced from a Self IP. If a floating Self IP exists, SNAT connections will be sourced from this IP address, otherwise they will be sourced from the non-floating address. Floating Self IP's are assigned to a Traffic Group and will failover between devices accordingly. This means that when a VirtualAddress is updated from Traffic Group 1 to 2, SNAT'd connections will now be sourced from a new IP address. The same potential concern exists for SNAT as with health monitors: if applications or firewalls accept only pre-configured IP addresses, we'll need to update them. However with SNAT an administrator might plan ahead with SNAT Lists. SNAT Lists are also assigned to Traffic Groups, and an administrator might create a SNAT List dedicated for an application. This SNAT List could be migrated between Traffic Groups at the same time as a Virtual Address and associated VM workload migration. Be mindful if updating the Traffic Group of a floating self IP or SNAT List: if multiple applications are expecting specific source IPs for health monitors or SNAT'd connections, those applications may need to be migrated together. Conclusion A stretched VLAN allows additional options for VM migrations. Careful planning of F5 BIG-IP deployments, using multiple Traffic Groups, will allow for a smooth migration plan. Thanks for reading!F5 AI Gateway - Secure, Deliver and Optimize GenAI Apps
AI has revolutionized industries by automating tasks, enabling data-driven decisions, and enhancing efficiency and innovation. While it offers businesses a competitive edge by streamlining operations and improving customer experiences, it also introduces risks such as security vulnerabilities, data breaches, and cost challenges. Businesses must adopt robust cybersecurity measures and carefully manage AI investments to balance benefits with risks. F5 provides comprehensive controls to protect AI and IT infrastructures, ensuring sustainable growth in an AI-driven world. Welcome to F5 AI Gateway - a runtime security and traffic governance solution
F5 Unveils New Built-In TCP Profiles
[Update 3/17: Some representative performance results are at the bottom] Longtime readers know that F5's built-in TCP profiles were in need of a refresh. I'm pleased to announce that in TMOS® version 13.0, available now, there are substantial improvements to the built-in profile scheme. Users expect defaults to reflect best common practice, and we've made a huge step towards that being true. New Built-in Profiles We've kept virtually all of the old built-in profiles, for those of you who are happy with them, or have built other profiles that derive from them. But there are four new ones to load directly into your virtual servers or use a basis for your own tuning. The first three are optimized for particular network use cases: f5-tcp-wan, f5-tcp-lan, and f5-tcp-mobile are updated versions of tcp-wan-optimized, tcp-lan-optimized, and tcp-mobile-optimized. These adapt all settings to the appropriate link types, except that they don't enable the very newest features. If the hosts you're communicating with tend to use one kind of link, these are a great choice. The fourth is f5-tcp-progressive. This is meant to be a general-use profile (like the tcp default), but it contains the very latest features for early adopters. In our benchmark testing, we had the following criteria: f5-tcp-wan, f5-tcp-lan, and f5-tcp-mobile achieved throughput at least as high, and often better, than the default tcp profile for that link type. f5-tcp-progressive had equal or higher throughput than default TCP across all representative network types. The relative performance of f5-tcp-wan/lan/mobile and progressive in each link type will vary given the new features that f5-tcp-progressive enables. Living, Read-Only Profiles These four new profiles, and the default 'tcp' profile, are now "living." This means that we'll continually update them with best practices as they evolve. Brand-new features, if they are generally applicable, will immediately appear in f5-tcp-progressive. For our more conservative users, these new features will appear in the other four living profiles after a couple of releases. The default tcp profile hasn't changed yet, but it will in future releases! These five profiles are also now read-only, meaning that to make modifications you'll have to create a new profile that descends from these. This will aid in troubleshooting. If there are any settings that you like so much that you never want them to change, simply click the "custom" button in the child profile and the changes we push out in the future won't affect your settings. How This Affects Your Existing Custom Profiles If you've put thought into your TCP profiles, we aren't going to mess with it. If your profile descends from any of the previous built-ins besides default 'tcp,' there is no change to settings whatsoever. Upgrades to 13.0 will automatically prevent disruptions to your configuration. We've copied all of the default tcp profile settings to tcp-legacy, which is not a "living" profile. All of the old built-in profiles (like tcp-wan-optimized), and any custom profiles descended from default tcp, will now descend instead from tcp-legacy, and never change due to upgrades from F5. tcp-legacy will also include any modifications you made to the default tcp profile, as this profile is not read-only. Our data shows that few, if any, users are using the current (TMOS 12.1 and before) tcp-legacy settings.If you are, it is wise to make a note of those settings before you upgrade. How This Affects Your Existing Virtual Servers As the section above describes, if your virtual server uses any profile other than default 'tcp' or tcp-legacy, there will be no settings change at all. Given the weaknesses of the current default settings, we believe most users who use virtuals with the TCP default are not carefully considering their settings. Those virtuals will continue to use the default profile, and therefore settings will begin to evolve as we modernize the default profile in 13.1 and later releases. If you very much like the default TCP profile, perhaps because you customized it when it wasn't read-only, you should manually change the virtual to use tcp-legacy with no change in behavior. Use the New Profiles for Better Performance The internet changes. Bandwidths increase, we develop better algorithms to automatically tune your settings, and the TCP standard itself evolves. If you use the new profile framework, you'll keep up with the state of the art and maximize the throughput your applications receive. Below, I've included some throughput measurements from our in-house testing. We used parameters representative of seven different link types and measured the throughput using some relevant built-in profiles. Obviously, the performance in your deployment may vary. Aside from LANs, where frankly tuning isn't all that hard, the benefits are pretty clear.F5 BIG-IP deployment with Red Hat OpenShift - keeping client IP addresses and egress flows
Controlling the egress traffic in OpenShift allows to use the BIG-IP for several use cases: Keeping the source IP of the ingress clients Providing highly scalable SNAT for egress flows Providing security functionalities for egress flowsExternal Monitor Information and Templates
This article is written by, and published on behalf of, DevCentral MVP Leonardo Souza. --- Introduction External monitors have been in use on F5 devices for a long time, and you can find a lot of code and articles about them here on DevCentral. However, most of those articles and code samples are very old and outdated. So, I think is a good time to update that with new information and new templates. In this article, I will provide some information about external monitors and templates you can use to build your own external monitor script and how you setup an external monitor. External Monitor The external monitor is a script that the F5 device will run to determine the status of a pool member (note: you can’t assign an external monitor to a node). The script runs the code and indicates if it was successful or not. The script indicates success outputting something to the standard output, and failure if nothing was outputted. The external monitor is your last resource in relation to monitoring, only to be used if there is no built-in monitoring that could be used to perform the monitoring. Built-in monitors are generally faster and easier to support as they do not require programming skills. External monitors run on Linux, while some built-in monitors can run on Linux or TMM. Linux is the kernel that CentOS uses, and CenOS is the base system that F5 uses, also known as the host system. TMM is the Kernel that F5 uses on TMOS, which is the F5 system. Running a built-in monitor in TMM is faster than in Linux. TMM monitors were introduced in version 13.1.0, for more information read this solution: https://support.f5.com/csp/article/K11323537 Configuration Steps Note: Instructions are for version 15.1.0, but should be similar in other versions, and you only need to setup this configuration in one device if you have an HA pair, as the configuration is shared during the config sync. Let’s create a simple external monitor to ping a pool member. You can do via tmsh, but it is simpler via GUI. Download the following file: https://raw.githubusercontent.com/leonardobdes/External_Monitor_Templates/master/bash_external_monitor_template.sh In the GUI go to System > File Management > External Monitor Program File List > Click Import. Select the file and give a name “simple_script” to the script, and import. That will import the file to the system, and make it available in the GUI. Now create a new monitor that points to the script file. In the GUI go to Local Traffic > Monitor > Click Create. Give the name “simple_monitor”, and select type as External. In the External Program select simple_script, click Finished. Now apply the new monitor to a pool member or pool. Here is the full configuration created: sys file external-monitor simple_script { checksum SHA1:817:01e135449c710e21dd090fdd688d940e319f97f5 create-time 2020-04-23:17:45:25 created-by admin last-update-time 2020-04-23:17:45:25 mode 33261 revision 1 size 817 source-path none updated-by admin } ltm monitor external simple_monitor { defaults-from external interval 5 run /Common/simple_script time-until-up 0 timeout 16 } ltm pool pool_asm { members { LABServerPHP:http { address 172.16.0.14 session monitor-enabled state up } test_server:http { address 172.16.0.19 session monitor-enabled state down } } monitor simple_monitor } The first part is the script imported to the system. Next, the monitor that was created. Lastly, a pool using that monitor, also showing that one pool member is up and another one is down. Templates You can write the script to be used in the external monitor in any language that is supported by the F5 devices. I created a template for each of the following languages: Bash Perl Python Tcl For the templates go to this codeshare link: External Monitor Templates | DevCentral On my speed tests, I got these times with the templates: Bash - 0.014 seconds Perl - 0.020 seconds Tcl - 0.021 seconds Python - 0.084 seconds As you can see, Bash is definitely the fastest. Perl and Tcl are very close, and Python is the slowest. But, you may need to use something that works very well in Python, so it might be your only option. Bash Template I will explain the Bash template, but the logic is the same for other programming languages. Each script has 2 versions, one with debug and one without, as indicated in their filename. I will explain the one with debug. This tells the system what to use to run the script. #!/bin/bash The script will be called with 2 arguments, first is the IP, and second the port. The IP is passed in IPv6 format, example “::ffff:172.16.0.14”, so you need to remove “::ffff:” Old scripts, and also the sample_monitor script that comes in the system, use “sed 's/::ffff://'`” that are very slow compared with the above. ip=${1:7} port=$2 This will create a log line in the file /var/log/ltm. logger -p local0.notice "$MON_TMPL_NAME - PID: $$ Name: $NODE_NAME IP: $ip Port: $port" This is where you add your code to test the pool member. This example is just a ping against the IP, and the “-c1” means a single packet. The script then saves the result to know if it failed or not. Next, it will log another line to /var/log/ltm. ping -c1 $ip &> /dev/null result=$? logger -p local0.notice "$MON_TMPL_NAME - PID: $$ Result: $result" Lastly, the scripts check if the result was successful. If successful, it will send up to the standard output. [ $result -eq 0 ] && echo "up" The log lines will be similar to these ones. Apr 24 13:59:15 LABBIGIP1.lab.local notice logger[5178]: /Common/simple_monitor - PID: 5177 Name: /Common/test_server IP: 172.16.0.19 Port: 80 Apr 24 13:59:18 LABBIGIP1.lab.local notice logger[5201]: /Common/simple_monitor - PID: 5200 Name: /Common/LABServerPHP IP: 172.16.0.14 Port: 80 Apr 24 13:59:18 LABBIGIP1.lab.local notice logger[5203]: /Common/simple_monitor - PID: 5200 Result: 0 Apr 24 13:59:18 LABBIGIP1.lab.local notice logger[5209]: /Common/simple_monitor - PID: 5177 Result: 1 Note that the monitoring tests can run in parallel, so the lines will not be sequential. Use the PID number (Process ID), to match the lines. Old scripts, and also the sample_monitor script that comes in the system, have multiple lines of code to check if the same monitor is still running. If running, the new process kills the old one, before performing the monitoring. I tested this behavior in version 11.3.0 and 15.1.0, and this is not necessary. Before starting the new process for the script, the system will kill the old one, so any code in the script to control that is unnecessary as it will not be used. I assume this was necessary for old versions, as I never tested, but is not necessary for currently supported versions. Arguments and Variables The external monitor has multiple settings, including the ones you will find in any other monitor, like interval and timeout. However, it has 2 extra settings, those are arguments and variables. Arguments are passed when calling the script, for example: bash simple_script.sh IP port argument3 Variables are set as environment variables. You can then access that variable in the script. Be aware that the system will add the variables from the monitor first, and then add some extra variables. The environment variables for version 11.3.0 are, with values as an example: MON_TMPL_NAME=/Common/bash_external_monitor_template ARGS_I=2 3 abc X=b PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/contrib/bin:/usr/local/bin:/usr/contrib/sbin:/usr/local/sbin:/usr/libexec NODE_PORT=80 PWD=/var/run SHLVL=1 NODE_IP=::ffff:172.16.0.19 NODE_NAME=/Common/test_server RUN_I=/Common/bash_external_monitor_template _=/bin/env ARGS_I will include the arguments you set in the monitor configuration, in this case, I have “2 3 abc”. X is the variable I added in the monitor configuration, with value “b”. That means you can use those variables in your script. Also, if you set the variable $PATH in your script as an example, it will get reset by the system, as the system variables are set after the monitor configuration variables. Let’s expand that with examples. Here is a monitor configuration with one argument and one variable. ltm monitor external simple_monitor { args 2 defaults-from external interval 5 run /Common/simple_script time-until-up 0 timeout 16 user-defined debug 1 } Here is the part of the script that changed: ping -c${3} $ip &> /dev/null result=$? [[ $debug -eq 1 ]] && logger -p local0.notice "$MON_TMPL_NAME - PID: $$ Result: $result Ping: ${3}" Now the script is using the argument 3 for the ping command, and the variable debug to define if log should occur. Let’s see the logs created: Apr 24 12:18:04 LABBIGIP1.lab.local notice logger[8179]: /Common/simple_monitor - PID: 8178 Name: /Common/test_server IP: 172.16.0.19 Port: 80 Apr 24 12:18:06 LABBIGIP1.lab.local notice logger[8196]: /Common/simple_monitor - PID: 8195 Name: /Common/LABServerPHP IP: 172.16.0.14 Port: 80 Apr 24 12:18:07 LABBIGIP1.lab.local notice logger[8206]: /Common/simple_monitor - PID: 8178 Result: 1 Ping: 2 Apr 24 12:18:07 LABBIGIP1.lab.local notice logger[8208]: /Common/simple_monitor - PID: 8195 Result: 0 Ping: 2 Ping is done with 2 packets, and the second log line still logged. Testing the Script External scripts are saved in this location: /config/filestore/files_d/Common_d/external_monitor_d That is for the Common partition, replace Common to the name of the partition if you import the script in another partition. The file will have an ID number (100479), and also a version number that changes when you update the file (19): :Common:simple_script_100479_19 That means you can go to that directory and run the script from there, and troubleshoot the errors. Don’t forget that when testing this way, you will not have the environment variables the system setup, you will have the environment variables from the shell where you run the command. Let’s test the simple simple_monitor discussed above. First, create the environment variables: export debug=1 MON_TMPL_NAME='/Common/simple_monitor' NODE_NAME='/Common/test_server' Run the script: ./\:Common\:simple_script_100479_22 '::ffff:172.16.0.14' 80 2 You will get the following log lines: Apr 24 16:03:48 LABBIGIP1.lab.local notice root[12206]: /Common/simple_monitor - PID: 12205 Name: /Common/test_server IP: 172.16.0.14 Port: 80 Apr 24 16:03:49 LABBIGIP1.lab.local notice root[12224]: /Common/simple_monitor - PID: 12205 Result: 0 Ping: 2 You need to escape the “:” with \. After you do the tests, delete the environment variables created: unset debug MON_TMPL_NAME NODE_NAME Conclusion I hope this article provides you all the information you need to create your own external monitor. After you understand how it works, it is very simple to use. Just don’t forget, no “echo down”, only output something when the monitor was successful, and make sure you suppress any command output.F5 BIG-IP and NetApp StorageGRID - Providing Fast and Scalable S3 API for AI apps
F5 BIG-IP, an industry-leading ADC solution, can provide load balancing services for HTTPS servers, with full security applied in-flight and performance levels to meet any enterprise’s capacity targets. Specific to S3 API, the object storage and retrieval protocol that rides upon HTTPS, an aligned partnering solution exists from NetApp, which allows a large-scale set of S3 API targets to ingest and provide objects. Automatic backend synchronization allows any node to be offered up at a target by a server load balancer like BIG-IP. This allows overall storage node utilization to be optimized across the node set, and scaled performance to reach the highest S3 API bandwidth levels, all while offering high availability to S3 API consumers. S3 compatible storage is becoming popular for AI applications due to its superior performance over traditional protocols such as NFS or CIFS, as well as enabling repatriation of data from the cloud to on-prem. These are scenarios where the amount of data faced is large, this drives the requirement for new levels of scalability and performance; S3 compatible object storages such as NetApp StorageGRID are purpose-built to reach such levels. Sample BIG-IP and StorageGRID Configuration This document is based upon tests and measurements using the following lab configuration. All devices in the lab were virtual machine-based offerings. The S3 service to be projected to the outside world, depicted in the above diagram and delivered to the client via the external network, will use a BIG-IP virtual server (VS) which is tied to an origin pool of three large-capacity StorageGRID nodes. The BIG-IP maintains the integrity of the NetApp nodes by frequent HTTP-based health checks. Should an unhealthy node be detected, it will be dropped from the list of active pool members. When content is written via the S3 protocol to any node in the pool, the other members are synchronized to serve up content should they be selected by BIG-IP for future read requests. The key recommendations and observations in building the lab include: Setup a local certificate authority such that all nodes can be trusted by the BIG-IP. Typically the local CA-signed certificate will incorporate every node’s FQDN and IP address within the listed subject alternate names (SAN) to make the backend solution streamlined with one single certificate. Different F5 profiles, such as FastL4 or FastHTTP, can be selected to reach the right tradeoff between the absolute capacity of stateful traffic load-balanced versus rich layer 7 functions like iRules or authentication. Modern techniques such as multi-part uploads or using HTTP Ranges for downloads can take large objects, and concurrently move smaller pieces across the load balancer, lowering total transaction times, and spreading work over more CPU cores. The S3 protocol, at its core, is a set of REST API calls. To facilitate testing, the widely used S3Browser (www.s3browser.com) was used to quickly and intuitively create S3 buckets on the NetApp offering and send/retrieve objects (files) through the BIG-IP load balancer. Setup the BIG-IP and StorageGrid Systems The StorageGrid solution is an array of storage nodes, provisioned with the help of an administrative host, the “Grid Manager”. For interactive users, no thick client is required as on-board web services allow a streamlined experience all through an Internet browser. The following is an example of Grid Manager, taken from a Chrome browser; one sees the three Storage Nodes setup have been successfully added. The load balancer, in our case, the BIG-IP, is set up with a virtual server to support HTTPS traffic and distributed that traffic, which is S3 object storage traffic, to the three StorageGRID nodes. The following screenshot demonstrates that the BIG-IP is setup in a standard HA (active-passive pair) configuration and the three pool members are healthy (green, health checks are fine) and receiving/sending S3 traffic, as the byte counts are seen in the image to be non-zero. On the internal side of the BIG-IP, TCP port 18082 is being used for S3 traffic. To do testing of the solution, including features such as multi-part uploads and downloads, a popular S3 tool, S3Browser, was downloaded and used. The following shows the entirety of the S3Browser setup. Simply create an account (StorageGRID-Account-01 in our example) and point the REST API endpoint at the BIG-IP Virtual Server that is acting as the secure front door for our pool of NetApp nodes. The S3 Access Key ID and Secret values are generated at turn-up time of the NetApp appliances. All S3 traffic will, of course, be SSL/TLS encrypted. BIG-IP will intercept the SSL traffic (high-speed decrypt) and then re-encrypt when proxying the traffic to a selected origin pool member. Other valid load balancer setups exist; one might include an “off load” approach to SSL, whereby the S3 nodes safely co-located in a data center may prefer to receive non-SSL HTTP S3 traffic. This may see an overall performance improvement in terms of peak bandwidth per storage node, but this comes at the tradeoff of security considerations. Experimenting with S3 Protocol and Load Balancing With all the elements in place to start understanding the behavior of S3 and spreading traffic across NetApp nodes, a quick test involved creating a S3 bucket and placing some objects in that new bucket. Buckets are logical collections of objects, conceptually not that different from folders or directories in file systems. In fact, a S3 bucket could even be mounted as a folder in an operating system such as Linux. In their simplest form, most commonly, buckets can simply serve as high-capacity, performant storage and retrieval targets for similarly themed structured or unstructured data. In the first test, we created a new bucket (“audio-clip-bucket”) and uploaded four sample files to the new bucket using S3Browser. We then zeroed the statistics for each pool member on the BIG-IP, to see if even this small upload would spread S3 traffic across more than a single NetApp device. Immediately after the upload, the counters reflect that two StorageGRID nodes were selected to receive S3 transactions. Richly detailed, per-transaction visibility can be obtained by leveraging the F5 SSL Orchestrator (SSLO) feature on the BIG-IP, whereby copies of the bi-directional S3 traffic decrypted within the load balancer can be sent to packet loggers, analytics tools, or even protocol analyzers like Wireshark. The BIG-IP also has an onboard analytics tool, Application Visibility and Reporting (AVR) which can provide some details on the nuances of the S3 traffic being proxied. AVR demonstrates the following characteristics of the above traffic, a simple bucket creation and upload of 4 objects. With AVR, one can see the URL values used by S3, which include the bucket name itself as well as transactions incorporating the object names as URLs. Also, the HTTP methods used included both GETS and PUTS. The use of HTTP PUT is expected when creating a new bucket. S3 is not governed by a typical standards body document, such as an IETF Request for Comment (RFC), but rather has evolved out of AWS and their use of S3 since 2006. For details around S3 API characteristics and nomenclature, this site can be referenced. For example, the expected syntax for creating a bucket is provided, including the fact that it should be an HTTP PUT to the root (/) URL target, with the bucket configuration parameters including name provided within the HTTP transaction body. Achieving High Performance S3 with BIG-IP and StorageGRID A common concern with protocols, such as HTTP, is head-of-line blocking, where one large, lengthy transaction blocks subsequent desired, queued transactions. This is one of the reasons for parallelism in HTTP, where loading 30 or more objects to paint a web page will often utilize two, four, or even more concurrent TCP sessions. Another performance issue when dealing with very large transactions is, without parallelism, even those most performant networks will see an established TCP session reach a maximum congestion window (CWND) where no more segments may be in flight until new TCP ACKs arrive back. Advanced TCP options like TCP exponential windowing or TCP SACK can help, but regardless of this, the achievable bandwidth of any one TCP session is bounded and may also frequently task only one core in multi-core CPUs. With the BIG-IP serving as the intermediary, large S3 transactions may default to “multi-part” uploads and downloads. The larger objects become a series of smaller objects that conveniently can be load-balanced by BIG-IP across the entire cluster of NetApp nodes. As displayed in the following diagram, we are asking for multi-part uploads to kick in for objects larger than 5 megabytes. After uploading a 20-megabyte file (technically, 20,000,000 bytes) the BIG-IP shows the traffic distributed across multiple NetApp nodes to the tune of 160.9 million bits. The incoming bits, incoming from the perspective of the origin pool members, confirm the delivery of the object with a small amount of protocol overhead (bits divided by eight to reach bytes). The value of load balancing manageable chunks of very large objects will pay dividends over time with faster overall transaction completion times due to the spreading of traffic across NetApp nodes, more TCP sessions reaching high congestion window values, and no single-core bottle necks in multicore equipment. Tuning BIG-IP for High Performance S3 Service Delivery The F5 BIG-IP offers a set of different profiles it can run its Local Traffic Manager (LTM) module in accordance with; LTM is the heart of the server load balancing function. The most performant profile in terms of attainable traffic load is the “FastL4” profile. This, and other profiles such as “OneConnect” or “FastHTTP”, can be tied to a virtual server, and details around each profile can be found here within the BIG-IP GUI: The FastL4 profile can increase virtual server performance and throughput for supported platforms by using the embedded Packet Velocity Acceleration (ePVA) chip to accelerate traffic. The ePVA chip is a hardware acceleration field programmable gate array (FPGA) that delivers high-performance L4 throughput by offloading traffic processing to the hardware acceleration chip. The BIG-IP makes flow acceleration decisions in software and then offloads eligible flows to the ePVA chip for that acceleration. For platforms that do not contain the ePVA chip, the system performs acceleration actions in software. Software-only solutions can increase performance in direct relationship to the hardware offered by the underlying host. As examples of BIG-IP virtual edition (VE) software running on mid-grade hardware platforms, results with Dell can be found here and similar experiences with HPE Proliant platforms are here. One thing to note about FastL4 as the profile to underpin a performance mode BIG-IP virtual server is that it is layer 4 oriented. For certain features that involve layer 7 HTTP related fields, such as using iRules to swap HTTP headers or perform HTTP authentication, a different profile might be more suitable. A bonus of FastL4 are some interesting specific performance features catering to it. In the BIG-IP version 17 release train, there is a feature to quickly tear down, with no delay, TCP sessions no longer required. Most TCP stacks implement TCP “2MSL” rules, where upon receiving and sending TCP FIN messages, the socket enters a lengthy TCP “TIME_WAIT” state, often minutes long. This stems back to historically bad packet loss environments of the very early Internet. A concern was high latency and packet loss might see incoming packets arrive at a target very late, and the TCP state machine would be confused if no record of the socket still existed. As such, the lengthy TIME_WAIT period was adopted even though this is consuming on-board resources to maintain the state. With FastL4, the “fast” close with TCP reset option now exists, such that any incoming TCP FIN message observed by BIG-IP will result in TCP RESETS being sent to both endpoints, normally bypassing TIME_WAIT penalties. OneConnect and FastHTTP Profiles As mentioned, other traffic profiles on BIG-IP are directed towards Layer 7 and HTTP features. One interesting profile is F5’s “OneConnect”. The OneConnect feature set works with HTTP Keep-Alives, which allows the BIG-IP system to minimize the number of server-side TCP connections by making existing connections available for reuse by other clients. This reduces, among other things, excessive TCP 3-way handshakes (Syn, Syn-Ack, Ack) and mitigates the small TCP congestion windows that new TCP sessions start with and only increases with successful traffic delivery. Persistent server-side TCP connections ameliorate this. When a new connection is initiated to the virtual server, if an existing server-side flow to the pool member is idle, the BIG-IP system applies the OneConnect source mask to the IP address in the request to determine whether it is eligible to reuse the existing idle connection. If it is eligible, the BIG-IP system marks the connection as non-idle and sends a client request over it. If the request is not eligible for reuse, or an idle server-side flow is not found, the BIG-IP system creates a new server-side TCP connection and sends client requests over it. The last profile considered is the “Fast HTTP” profile. The Fast HTTP profile is designed to speed up certain types of HTTP connections and again strives to reduce the number of connections opened to the back-end HTTP servers. This is accomplished by combining features from the TCP, HTTP, and OneConnect profiles into a single profile that is optimized for network performance. A resulting high performance HTTP virtual server processes connections on a packet-by-packet basis and buffers only enough data to parse packet headers. The performance HTTP virtual server TCP behavior operates as follows: the BIG-IP system establishes server-side flows by opening TCP connections to pool members. When a client makes a connection to the performance HTTP virtual server, if an existing server-side flow to the pool member is idle, the BIG-IP LTM system marks the connection as non-idle and sends a client request over the connection. Summary The NetApp StorageGRID multi-node S3 compatible object storage solution fits well with a high-performance server load balancer, thus making the F5 BIG-IP a good fit. S3 protocol can itself be adjusted to improve transaction response times, such as through the use of multi-part uploads and downloads, amplifying the default load balancing to now spread even more traffic chunks over many NetApp nodes. BIG-IP has numerous approaches to configuring virtual servers, from highest performance L4-focused profiles to similar offerings that retain L7 HTTP awareness. Lab testing was accomplished using the S3Browser utility and results of traffic flows were confirmed with both the standard BIG-IP GUI and the additional AVR analytics module, which provides additional protocol insight.
