F5 Distributed Cloud and Amazon FSx for NetApp ONTAP - Global Replication and Anywhere Access

F5 Distributed Cloud (XC) is a SaaS solution which securely exposes services to the correct service consumers, whether the endpoints of the communications are in public clouds, private clouds, or on-prem data centers. In many cases, distributed load balancers are tactical XC tools to proxy published services, like a web application on TCP port 443 to backend, secured origin pools. In other cases, a nimble layer 3 service, along the lines of a next-generation multi-site turnkey VPN solution are equally attractive, where protocols like SnapMirror might consume a small range of TCP ports to empower backup and replication jobs between enterprise endpoints.

This article explores how to easily set up a replication solution for islands of NetApp storage, such as ONTAP clusters located at enterprise offices and data centers. The end state is all critical data securely replicated to the cloud-native Amazon FSx for NetApp ONTAP offering. Beyond backup and recovery purposes, the solution may in turn allow local AWS-instantiated read/write access to key global volumes with the highest possible QoS guarantees. Such a centralized corralling of critical data might align with feeding corporate AI RAG initiatives that infuse LLMs with private corporate data sources to reach optimal context-specific AI outcomes.

 

Technology Overview – Leverage Secure Layer Three Multicloud Connections

The XC Network Connect module harnesses the built-out global F5 infrastructure to provide automatic reachability between customer edge (CE) sites on-prem, in private cloud or enterprise tenants within an array of public cloud providers. Where WAN facilities may already exist, CE sites can also be directly inter-connected with those data planes. A key deliverable is that routing is automatic and those simple troubleshooting and analytic tools required by NetOps are uniform and available as a single-pane-of-glass experience in the XC SaaS console.

Network segmentation is essential in controlling modern and complex enterprise environments. Traditional methods like Virtual Routing and Forwarding (VRFs) and Multiprotocol Label Switching (MPLS) have long been used to create isolated network segments in on-prem setups. F5 XC ensures segmentation in environments like AWS, and it can extend the same segmentation to on-prem environments. These techniques separate traffic, enhance security, and improve network management by preventing unauthorized access and minimizing the attack surface.

Any interface on a CE node, such as in an office computing room, may be associated with an XC “segment”, essentially a VRF setup. For instance, segments might be labeled production, development, and quality assurance. Distributed Cloud now supports up to eight Ethernet interfaces on physical or virtual appliances, along with VLAN-tagged sub-interfaces. As such, the number of segments required is not normally a concern. Azure VNETs may also be included in segments.

With AWS, transit gateway (TGW) sites are particularly useful for high scalability, as they allow a hub-and-spoke architecture for routing decisions. As such, just one single CE site in a TGW virtual private cloud (VPC) can enable reachability to countless segments. The following is a simple example of four segments simply labeled as color names, including a CE site in the lower right with attachments to multiple segments.

It is common to require interconnectivity between segments, in other words, to “leak” routes automatically between selected segments in certain scenarios. A use case might be a shared VPC where modern logging receivers and analytics solutions reside. In the following simple case, it can be seen that both production and development segments, which span AWS and on-prem CE sites, may need to interact with the shared VPC.

The mentioned common reachability requirement, allowing shared services such as logging to interact with other segments, does not require any routing knowledge or command-line tools. A simple “segment connector” GUI within the XC console allows simplified workflows.

In the following XC screenshot, two existing segments are selected for mutual reachability, highlighted options are direct connect or to perform a SNAT operation to adjust source IP addresses.

 

Replicating Global NetApp Volumes to Amazon FSx for NetApp ONTAP

Amazon FSx for NetApp ONTAP, or simply FSx for ONTAP for brevity, allows a range of benefits to enterprise, including true multiprotocol support for accessing data, including NFS, SMB, S3, iSCSI, and NVMe-over-TCP protocols. This AWS-based NetApp service offering also benefits from scalability control. One may support their growing data sets with storage capacity that grows and shrinks automatically, no day-to-day manual intervention required. One of the principal use cases is controlled migration of apps to cloud environments; popular choices in moving some applications to AWS include highly scalable web applications, data-intensive applications requiring large storage and processing power, and real-time applications with high traffic; applications with complex data analytics needs.
One of the differentiating features of FSx for ONTAP is that beyond being a scalable cloud-hosted storage solution, it integrates very tightly with long-established on-prem NetApp solutions, key NetApp workflows include:

• Back up, archive, or replicate data from your on-premises file servers to FSx for ONTAP, harnessing mature SnapMirror technology to simplify business continuity and meet modern data retention and disaster recovery requirements.
• Data in an on-premises NetApp file system that requires access or processing from AWS hosts or services with ultra-low latency, configure FSx for ONTAP as an in-cloud cache for your valuable on-premises data by using NetApp FlexCache. When used as a cache, FSx for ONTAP provides lightning-fast access to your popular on-premises data sets from AWS compute instances.
• Embrace cloud cost efficiencies: Unlike generally fixed costs with growing on-prem deployments that may be overprovisioned, pay only for the resources you actually use, with various pricing models to optimize costs.

F5 Distributed Cloud will serve as the underlay, turn-key networking solution to bind NetApp elements, virtually anywhere around the globe, with FSx for ONTAP. The following is a simple deployment shown visually, with the intention of SnapMirroring key enterprise volumes around the world to FSx for ONTAP instantiated on AWS East-2.

Replicate AI Training Data from On-Prem to AWS FSx and Locally Utilize with EC2 Compute

A key rationale for replicating data from on-prem to AWS is disaster recovery. A typical architecture selected for FSx for ONTAP is to spread highly-available volumes across multiple availability zones (AZ) for the utmost in redundancy. In this example, a volume representing files used in a RAG AI document volume is replicated to AWS. This end goal can be achieved in many approaches, including NetApp BlueXP console, or using the local AWS console and on-prem NetApp System Manager to create the replication. In this particular case, a simple command-line approach is utilized and can be found documented here.

The source NetApp ONTAP Select is located in the Seattle area and a requirement exists to Snap Mirror volumes to FSx for ONTAP. With a pair of F5 CE nodes, the red dashed line above indicates the required L3 reachability. With CE sites deployed, the configuration of the production segment is done on the following screen in the Multicloud Network Connect module of the XC console.

In the above, one notes the production segment consists in total of two VPCs and two physical locations, Seattle (Redmond, WA) and San Jose. The “Allow traffic from this segment to Internet” indicates hosts on the attached subnets will be permitted outbound Internet access, in much the same way a NAT Gateway can optionally facilitate this in many common AWS deployments.
Looking at the NetApp traditional System Manager Volumes view, we isolate the volume which we wish to replicate to FSx for ONTAP, RAG_Secure_Documents.

Through the FSx for ONTAP command interface, we start by creating a volume in the AWS offering to receive replicated content. This is a data protection volume as denoted by type “DP”:

FSx-Dest::> vol create -vserver fsx -volume RAG_Secure_Documents -aggregate aggr1 -size 1g -type DP -tiering-policy all

We then establish cluster peering between the on-prem ONTAP Select and the FSx for ONTAP offering, through similar commands on each end:

FSx-Dest::>  cluster peer create -address-family ipv4 -peer-addrs 10.50.0.221
Seattle ONTAP::> cluster peer create -address-family ipv4 -peer-addrs 10.1.200.36,10.1.200.202

After peering, the storage virtual machines (SVMs) that govern the volumes, both on-prem and in AWS, can then establish the SnapMirror relationship and transfer the data to replicate our volume, via F5 XC, between Seattle and FSx for ONTAP.

FSx-Dest::>snapmirror create -source-path svm0:RAG_Secure_Documents -destination-path fsx:RAG_Secure_Documents -vserver fsx -throttle unlimited
FSx-Dest::>snapmirror initialize -destination-path fsx:RAG_Secure_Documents -source-path svm0:RAG_Secure_Documents

The following command, applied against FSx for ONTAP, demonstrates a successful and on-going SnapMirror relationship from on-prem to AWS for the volume “RAG_Source_Documents_2024”:

FsxId05954ec2335664e17::> snapmirror show -fields state,status
source-path                                        destination-path                       state              status
-------------------------                          ------------------------                 ----------         -----
svm0:RAG_Secure_Documents           fsx:RAG_Secure_Documents         Broken-off      Idle
svm0:RAG_Source_Documents_2024 fsx:RAG_Source_Documents_2024 Snapmirrored  Idle

The fact that the other volume, RAG_Secure_Documents has been broken off stems from the fact that this volume in FSx for ONTAP has been flipped from a data protection (DP) volume to an actively used read-write (RW) volume. Perhaps EC2 instances that are heavily steeped in AI workloads can benefit from AWS-housed volumes and a single-point of reference for global islands of NetApp volumes centralized to FSx for ONTAP for fixed-length projects.

In this case, a standard NFS mount command will allow any EC2 host to consume the data:

The above are just two examples of the multi-faceted value of corralling enterprise NetApp data volumes with F5, SnapMirror for the purposes of replication and disaster preparedness, coupled with the value of centralized, read-write copies of valuable data.

FlexCache – High Performance Use of Worldwide NetApp Volumes centrally Accessed via F5 and FSx for ONTAP

Caching is a valuable longstanding tool to make content that might otherwise travel across WANs, including trans-continental links, appear local, with staleness checks occurring automatically to validate the latest files are always used. One of the many scenarios potentially leveraging NetApp’s FlexCache technology is explored in this section.

Consider workflows that are housed in AWS East-2, based in Ohio that make frequent use of files physically housed on a west coast ONTAP cluster in the Seattle area, in Redmond, Washington. F5 Distributed Cloud allows for seamless interconnections of enterprise data volumes, however, there is much to be gained by AWS East-2 EC2 hosts consuming local versions of the west coast content whenever possible. The laws of physics impose constraints that even modern fiber-optic networks must fall within. The distance between Seattle and Columbus, Ohio is approximately 3,200 km (almost 2,000 miles) and in 2025 on-going measurements of ICMP response times put latency in the 60 millisecond region (link).

Trans-oceanic latency will be even more substantial and NAS access protocols like Server Message Block (SMB) are notoriously chatty, with end-to-end application message pairs required to enable even simple, single-file copy jobs. All popular NAS protocols, regardless of application versions, ride upon TCP, which itself has an ongoing round-trip acknowledgment tax, which imposes a higher latency penalty for long WAN connections.

With FlexCache volumes configured in a central location, where end user consumption is highest, in our example AWS East-2, the first read of files from the west coast will be bound by the above laws of physics. However, subsequent reads by users of the same files will be served out of the FlexCache volume in FSx, thereby making load or copies times negligible for “hot” content.

To demonstrate this, 25 megabytes of Gutenberg Project free-access novels were uploaded via NFS to the Redmond ONTAP Select appliance via a local client. The following FlexCache volume was created on FSx for ONTAP, following the published guidelines for AWS FSx found here.

(at FSx command line):

vserver peer create -vserver svmfsx0 -peer-server svm2 -peer-cluster f5netappclusterE -local-name svm2 -application flexcache

(at west coast ONTAP Select command line):

vserver peer accept -vserver svm2 -peer-vserver svmfsx0 -local-name svmfsx0

(at FSx command line):
volume flexcache create -vserver svmfsx0 -size 15g -volume CacheVol -origin-volume seattleofficefiles -origin-vserver svm2 -junction-path /flexcache -aggr-list aggr1

 

The above screenshot demonstrates the result of the commands, a new volume “CacheVol” associated with one of the FSx for ONTAP storage virtual machines and accessible to EC2 or other hosts at the path “/flexcache”. The NetApp FlexCache inner workings, including discussions around the RAL protocol, which underpins cache management, are discussed here.
A test EC2 host in AWS East-2 has mounted the FlexCache volume “CacheVol” using the following standard command sequence.
# sudo mkdir -p /seattle-files
#sudo mount -t nfs -o nfsvers=3 10.1.200.104:/flexcache /seattle-files

The 25-megabytes of books are copied to a local folder on the EC2 with the following command.
# cd /seattle-files
# cp *.epub /local-file-copies

The resulting network activity, copying the remote files to the EC2 while simultaneously automatic loading of the CacheVol, is seen through the F5 XC Flow Analyzer, which has been set to report on a 10-minute window of traffic leaving the Seattle CE site (s-gorman-redmond-smsv2-ce) and targeting the AWS East-2 Transit Gateway (TGW) CE site (s-gorman-tgw-may25).

One can see the data totals reflect the large file transfer, being close to 25 megabytes, however the TCP connections as expected do not use the NAS access protocol port, typically TCP port 2049. Instead, TCP 11105, the well-known port used traditionally by SnapMirror, is leveraged. This reminds one that block-level optimizations like data compression will be enacted. Multiple connections are made use of, client-side ephemeral ports as seen in this case are in the 20444 and up range.

A few minutes later, the EC2 host was instructed again to copy these publications to another, different local folder. In this case, the copy process was virtually instantaneous from the user’s perspective. The 25 megabytes of files were immediately seen in the local folder. Analyzing the F5 Flow Analysis tool for the same amount of time, ten minutes, covering the span of this second copy, we see negligible traffic crossing the continent; caching has proven its value.

The F5 flow analyzer is a nice, simple asset to other integrated tools such as traceroute or TCPDump; for instance the flow analyzer can report on individual IP addresses with top talkers or top listeners isolated over 5 minutes, 1 hour, or 24 hours all available.

Summary

In this article, we have demonstrated that the F5 Distributed Cloud Network Connect module, a modern approach to layer 3 multicloud networking (MCN), can be the turnkey glue for global NetApp storage elements; a simple networking end-state that is both performant and heightens security through segmentation. In alignment with this is the simple cloud-backed NetApp storage solution Amazon FSx for NetApp ONTAP, that uses well-known NetApp workflows to setup both NAS (NFS, SMB) and SAN (iSCSI) access. It concurrently offers a consolidated suite of storage management tools such as disaster recovery for islands of traditional ONTAP appliances throughout an enterprise’s global footprint.

Three principal use cases were demonstrated in this article. First, a disaster recovery approach through on-going SnapMirror sessions from remote sites to a centralized AWS FSx cloud-native highly available solution. Second, the data protection volumes created through SnapMirror were easily re-purposed to offer classic read-write volumes accessible through standard NAS protocols.

A final use case was the ability to enable NetApp FlexCache volumes to provide lightning-quick access to distant volumes through F5 XC MCN. As data from offices, such as a west coast Seattle site, traveled towards consumption, for example east coast EC2 instances within AWS, FlexCache quickly populated the FlexCache volumes within FSx for ONTAP. Any subsequent access requests to this West Coast data were observed to be provided through local, ultra-low-latency access times typical of intra-AWS region communications. F5 has provided a set of console views in the XC SaaS interface to monitor the underlying NetApp flows as they pass through the Distributed Cloud infrastructure.