F5 App Connect and NetApp S3 Storage – Secured Scalable AI RAG
F5 Distributed Cloud (XC) is a SaaS solution which securely exposes services to the correct service consumers, whether the endpoints of the communications are in public clouds, private clouds, or on-prem data centers. This is particularly top of mind now as AI RAG implementations are easy to set up but are really only effective when the correct, often guarded, enterprise data stores are consumed by the solution. It is a common scenario where the AI compute loads are executed in one location, on-prem or perhaps in a cloud tenant. However, the data to be ingested, embedded, and stored in a vector database to empower inferencing may be distributed through many different geographies. The data sources to be ingested into RAG are often stored in NetApp form factors, for instance StorageGrid, a native object-first clustered solution for housing S3 buckets. Also, the ONTAP family, where frequently files are accessed through NAS protocols like NFS or SMB, today can see the RAG source content exposed as objects through S3-compliant API calls and the corresponding protocol license. Technology Overview The XC module App Connect leverages L4-L7 distributed load balancers to securely provide a conduit to enterprise NetApp-housed data for the centralized AI workloads leveraging RAG. The following is the setup objective for this article, although many customer edge (CE) sites exist, we aim to bring together corporate documents (objects) in a San Jose, California datacenter to a self-hosted AI/RAG solution running in a Seattle area datacenter. The Advantage of Load Balancers to Securely Attach to Your Data Previous articles have leveraged the XC Network Connect module to bring together elements of NetApp storage through NAS protocols like NFS in order to run AI RAG workloads, both self-hosted and through secure consumption of Azure OpenAI. The Network Connect module provides secure L3 (IP) connectivity between disparate sites. An advantage is Network Connect will support all IP-based protocol transactions between sites, and firewall rules to preclude unwanted traffic. Network Connect is great when ease of deployment is paramount, however if you know the protocols to be supported are HTTP or TCP-based read on about App Connect, a solution that can address any IP overlap that may exist between your various sites to be interconnected. App Connect is a different take on providing connectivity. It sets up a distributed load balancer between the consumer (in our case an AI server running LLMs and a vector database) and the services required (NAS or S3 accessible remote data stores on NetApp appliances or hosted services). The load balancer may be an HTTPS offering, which allows the leveraging of F5’s years of experience in web security solutions, including an industry-leading web application firewall (WAF) component. For non-web protocols, think NFS or SMB, a TCP layer load balancer is available. Advantages are that only the subnets where the consumer exists will ever receive connectivity and advertisements around the service configured. The load balancer can also expose origin pools that are not just private IP addressed appliances, origin pools can also be Kubernetes services. A final App Connect feature that is noteworthy, the offering provided is an L4 through L7 service (such as HTTPS), and as such the local layer 3 environment of the consumer and, in our case, storage offering is irrelevant. A complete overlap of IP, perhaps both ends are using the same 10.0.0.0/16 allotments, is acceptable, something extremely valuable within divisions of large corporations that have separately embraced similar RFC-1918 address spaces. Also, perhaps through mergers and acquisitions the net result in a major institution are widespread instances of duplicate IP spaces in use, IP renumbering projects are legendary as they are lengthy and fraught with the risks of touching core routing tables. Lastly, applications that require users to configure IP addresses into GUIs are problematic as values are dynamic, App Connect provides services by name typically and this is less burdensome for IT staff who manage applications. A Working AI RAG Setup Using NetApp StorageGRID and XC App Connect A Ubuntu 22.04 Linux server was configured as a hosted LLM solution in a Seattle-area datacenter. The Ollama open-source project was installed in order to serve both generative AI LLMs quickly (llama3.1, mistral and phi3 were all used for comparative results) and the required embedding LLM. The latter LLM is needed to create vector embeddings of both source enterprise documents and subsequent real-time inference query payloads. Through semantic similarity analysis, RAG will provide augmented prompts with useful and relevant enterprise data to the Ollama-served models for better AI outcomes. Using the s3fs offering on Linux one can quickly mount s3 buckets as file systems using FUSE (file system in user space). The net result is that any S3 bucket, supported natively by NetApp StorageGRID and through a protocol license on ONTAP appliances, can now be mounted as a Linux folder for your RAG embedding pipeline to build a vector database. The key really is how to easily tie together S3-compliant data sources through your modern enterprise, no matter where they exist and the form-factor they are in. This is where XC App Connect enters the equation, dropping a modern distributed load balancer to project services across your network locations. The first step in configuring the HTTPS Load balancer to connect sites is to enter the Multi-Cloud App Connect module of the XC console. Once there, primarily three key items need to be configured: An origin pool that points at the StorageGrid nodes or the local load balancer sitting in front of the nodes, these are private addresses within our San Jose site. An HTTPS load balancer that ties a virtual service name (in our case the arbitrary name s3content.local), to our origin pool. Establish where the service name will be projected by DNS and connectivity will be allowed, the service s3content.local is not to be projected into the global DNS but rather will only be advertised by the Seattle CE inside interface, essentially making this load balancer a private offering. Here is the origin pool setup, in our case a BIG-IP is being used as a local load balancer for StorageGRID and thus its private San Jose datacenter address is used. To achieve the second item, an HTTPS load balancer, we key in the following fields, including the FQDN of the service (s3content.local), the fact that we will provide a TLS certificate/key pair to be used by the load balancer, and the one-check option to create an HTTP-to-HTTPS redirection service too. Lastly, the advertisement of our service will only be supported by the CE node at the Seattle site, requests for s3content.local from our AI server will resolve to the local CE node inside network interface IP address. The App Connect load balancer will ensure the underlying connectivity, through the underlay network, to the origin pool (StorageGRID) in San Jose. RAG Using Ollama-Based Models and Remote StorageGRID Content Various methods exist to leverage freely available, downloadable AI LLMs. One popular approach is huggingface.com whereas another can be to leverage the Ollama framework and download both embedding and generative models from ollama.com. The latter approach was followed in this article and in keeping with past explorations, Python 3 was used to manipulate AI programmatically, including the RAG indexing tasks and the subsequent inferencing jobs. Ollama supports a Docker-like syntax when used interactively from the command line, the one embedding model and three generative models are seen below (from Ubuntu 22.04 terminal). $ ollama ls NAME ID SIZE MODIFIED llama3.1:latest 42182419e950 4.7 GB 6 days ago mistral:latest 974a74358d6 4.1 GB 6 days ago nomic-embed-text:latest 0a109f422b47 274 MB 6 days ago phi3:latest 4f2222927938 2.2 GB 7 days ago The RAG tests includes ingestion of both .txt and .pdf documents provided by App Connect from NetApp StorageGRID. A private CA certificate was created using an OpenSSL-derived tool and loaded into the Seattle Linux and Windows hosts. That CA cert was then used to create a pkcs12-packaged certificate and key set for s3content.local and uploaded to the HTTPS load balancer setup on Distributed Cloud. A quick Windows-based S3browser test confirmed reachability from Seattle: To leverage RAG document ingestion on Linux, the s3fs project was made use of. A Linux folder can be mounted using NAS protocols like NFS, a simple example might be: #sudo mount -t nfs 10.50.0.202:/f5busdev /home/steve/f5bizdev/ Using s3fs, folders can similarly be mounted which tie back to buckets on the remote StorageGRID: #s3fs mybucket001 /home/steve/rag-files-001 -o url=https://s3content.local:443 -o use_path_request_style -o passwd_file=/home/steve/.passwd-s3fs At this point RAG ingestion can take place. The net effectiveness of RAG is often attributed to the quality of the embedding LLM used. In this case the Python scripts leveraged the Ollama API and the locally downloaded “nomic-embed-text” model, one that is generally considered just passable but is small and runs well on CPU-only hosts. The Gutenberg project novel The Great Gatsby by F. Scott Fitzgerald was consumed from the NetApp remote bucket and a detailed question was asked of llama3.1, first without RAG’s augmentations, meaning just the pre-trained data was leveraged. $ python search.py What university did Gatsby attend? Answered without RAG: “In the novel "The Great Gatsby" by F. Scott Fitzgerald, the character of Jay Gatsby does not reveal where he attended university, as it is never explicitly stated in the book. However, it has been speculated and suggested in various adaptations and interpretations that he may have studied at St. Olaf College in Minnesota or Yale University due to some similarities with their campus architecture in the novel. But again, this remains purely speculative and not confirmed by Fitzgerald himself.” Note, the user prompt shown (nor the system prompt) told llama3.1 that the question pertained to a specific novel, it figured that part out on its own based on the impressive base training that is baked into foundational models. However, the answer is imperfect. With RAG supporting the query the result was much better; it mimics a response of an individual who had actually just read the entire novel, cover-to-cover, and is able to provide more nuanced and accurate answers (trimmed for brevity). $ python search.py What university did Gatsby attend? Answered with RAG: "According to the text, Gatsby attended 'Oggsford College' in England, not Oxford University. It is also mentioned later in the text that Gatsby told Mr. Wolfshiem that he had been employed by Dan Cody on a boat, and that this occurred after World War I, but before he claimed to have attended Oggsford College (which suggests that his attendance at the college may be questionable)." Pointing Linux mounted folders towards buckets containing pdf documents seemed to work best when the source documents were smaller. For instance, user manuals worked well, even though the embeddings focused upon textual chunks and disregarded diagrams. This script was instructed to provide attributions within the augmented text provided to the LLM, specifically the manual page number and document chunk number from that page. The following is a result using the smallest Ollama generative LLM test, the Phi3 model and quizzing it about some lawn maintenance equipment. $ python query_data.py "Are there any concerns with the muffler on this lawn mower?" Response with RAG: Yes, according to the given context, there is a concern about the muffler on this lawn mower. It is stated that if the engine has been running, the muffler will be hot and can severely burn you. Therefore, it is recommended to keep away from the hot muffler. Sources: ['../rag-files-003/toro_lawn_mower_manual.pdf:12:2', '../rag-files-003/toro_lawn_mower_manual.pdf:12:1', '../rag-files-003/toro_lawn_mower_manual.pdf:21:1', The findings were less positive with very large PDF documents. For instance, Joseph Hickey’s 1975 book A Guide to Bird Watching is available in the public domain and totals almost 300 pages and is 32 megabytes in size. Regardless of the LLM, mistral or llama3.1 included, rarely were questions taken directly from specific pages answered with precision. Questions supported by statements buried within the text, “Where was the prairie chicken first observed?” or “Have greenfinches ever been spotted in North America and how did they arrive?” all went unanswered. Get Better Mileage from Your RAG To optimize RAG, it’s unlikely the generative LLMs are at fault; with Ollama allowing serial tests it was quickly observed that none of Mistral, Llama3.1 or Phi3 differed when RAG struggled. The most likely route to improve responses is to experiment with the embedding LLM. The ability to derive semantic meaning for paragraphs of text can vary, Hugging Face provides a leaders board for embedding LLMs with their own appraisal via a performance scoring system in the massive text embedding benchmark (MTEB). Other ideas are to use significantly larger chunk sizes for large documents, to reduce the overall number of vector embeddings being semantically compared, although a traditional 2,048 token context window in inferencing would limit how much augmented text can be provided per RAG-enabled prompt. Finally, multiple ways exist to actually choose similar vector embeddings from the database, approaches like cosine similarity or Euclidean distance. In these experiments, the native feature to find “k” similar vectors is provided by ChromaDB itself, as explained here. Other methods that play this critical search for related, helpful content would include Facebook AI Similarity Search (FAISS), which uncouples the search feature from the database, reducing risks of vector DB vendor lock in. Other libraries, such as the compute-cosine-similarity library are available on-line, including support for languages like JavaScript or TypeScript. Future experiments with better embedding models or possibly changing document chunking sizes to larger values might well produce even better results when scaling up your RAG deployment to enterprise scale and the expected larger documents. F5 XC App Connect for Security and Dashboard Monitoring The F5 Distributed Cloud load balancers provide a wealth of performance and security visibility. For any specific time range, just glancing at the Performance Dashboard for our HTTPS load balancer quickly gives us rich details including how much data has been moved to and from NetApp, the browser types used, and specifically which buckets are active on the StorageGRID. Enterprises frequently invest heavily in dedicated monitoring solutions, everything from raw packet loggers to application performance measurement (APM) tools, sometimes offering PCAP exports to be consumed in turn by other tools such as Wireshark. Although Distributed Cloud load balancers are a secure communications solution, a wealth of monitoring information is available. Both directions of transactions, consisting of requests coupled with their responses as a single entity, are monitored and available in both a decoded, information pane and a rawer .json format for consumption by other tools. Here is one S3 object write from Seattle, crossing the XC HTTPS load balancer, and storing content on the San Jose StorageGRID. The nature of the client, including browser agent (S3 Browser in the image) and TLS signature for this client are available, as well as the object name and bucket it was targeting on the NetApp appliance. Useful Approaches for Managing and Securing Storage API Traffic A powerful module in Distributed Cloud that locks in on NetApp traffic, in this case carried between cities by S3-compliant APIs, is the API Discovery module. As seen in the following image, an operator can add approved API endpoints to the “Inventory”, similar to adding the endpoint to a swagger file, something easily exported from the GUI and potentially integral in enterprise API documentation. As denoted in the next image, the remaining “Shadow” endpoints that were all automatically discovered, allow quick attention brought to them and is an enabler of a positive security approach whereby shadow traffic could be blocked immediately by the load balancer. In this case, a quick method of blocking unsanctioned access to specific StorageGRID buckets is arrived at. Also worth noting in the above screenshot, the most active APIs for a selected time period, 5 minutes to a full day perhaps, are brought to the operator’s attention. Finally, a last example of the potential value of the API discovery module are the sensitive data columns. Both custom provided data types observed in flight (such as phone numbers or national health ID values) as well as non-compliance to industry-wide guidelines (such as PCI_DSS) are flagged per S3 API endpoint. Automatic Malicious User Mitigation Distributed Cloud also offers a valuable real-time malicious user mitigation feature. Using behavioral observations, many harnessing AI itself, rogue consumers or generators of S3 objects can be automatically blocked. This may be of particular use when the distributed HTTPS load balancer provides NetApp S3 access to a wider range of participants, think of perhaps partner enterprises with software CE sites installed at their location. Or, ramping up, consider general Internet access where the App Connect load balancer projects access to FQDNs through global DNS and the load balancer is instantiated on an international network of regional edge (RE) sites in 30+ major metropolitan markets. This user mitigation feature can be enacted by first tying a user identity policy to the HTTPS load balancer. One valuable approach is to make use of state-of-the-art client-side TLS fingerprinting, JA4 signatures. Combining TLS fingerprinting with other elements, such as client source IP, can assist is categorizing the unique user driving any attempted transaction. With this selection in place, an operator need only flip “on” the automatic mitigations and security will be ratcheted up for the load balancer. As seen in the following screenshot, XC and its algorithms can gauge the threat level presented by users and respond accordingly. For detected low threat levels JavaScript challenges can be presented to a client browser, something not requiring human intervention but assists in isolating DDoS attackers from legitimate service clients. For behavior consistent with medium threat levels, something like a Captcha Challenge can be opted for, in cases where a human should be driving interactions with the service presented by the load balancer. Finally, upon the perception of high threat levels Distributed Cloud will see to it that the user is placed into a temporarily blocked state. Summary In this article we have demonstrated that the F5 Distributed Cloud App Connect module can be set up to provide a distributed L4-7 load balancer that can bring remote islands of storage, in this case NetApp appliances, to a centralized RAG AI compute platform. Although TCP-based NAS protocols like NFS could be utilized through TCP load balancing, this particular article focused upon the growing S3-compatible API approach to object retrieval, which uses HTTPS as transport. Through a pair of CE sites, an S3 service was projected from San Jose to Seattle, the origin pool was a cluster of NetApp StorageGRID nodes, and the consuming entity was Ubuntu running Ollama LLMs models in support of RAG. The service was exclusively projected to Seattle and the net result was AI outcomes that consumed representative novels, research guides and product user manuals for contextually meaningful responses. The App Connect module empowers features such as rich transactional metrics, API discovery coupled with enforcement of S3 bucket access, and finally the ability to auto-mitigate worrisome users with reactions in accordance with threat risk levels.Migrating between sites using stretched VLANs
Background Recently I was asked the following question by a customer: I have 2 physical datacenters. Datacenter A is a legacy datacenter and almost all workloads run here, and Datacenter B is a new location with almost no workloads deployed. I must migrate quickly out of location A into B, in order to close location A. Using NSX-T, I have "stretched" VLAN's between sites, meaning that a single VLAN and CIDR block is available to VM's in either datacenter. I have 2x F5 BIG-IP devices configured in an HA pair (Active/Standby) in location A, but none yet in B. I have thousands of Virtual Servers configured on my BIG-IP devices, and many thousands of VM's configured as pool members, all running in location A. I can migrate workloads by replicating VM's across datacenters, leaving their IP addresses unchanged after migration to location B. What are my options for migrating the BIG-IP devices to location B?I'd like to maintain High Availability within each datacenter, minimize any traffic between the sites, and minimize operational difficulties. Let's take a look at these requirements and review potential solutions. Defining our migration Firstly, let's define our existing environment and challenges in a little more detail. What is a stretched VLAN? "Stretched VLAN" is a shorthand phrase for the practice of extending Layer 3 networks across physical sites. This is useful in situations like VM migration across data centers without needing to change the VM's network configuration. If datacenters are within a few miles of each other, direct Layer 2 connectivity may be possible. A more commonly preferred approach is tunneling Layer 2 networks across routed (Layer 3) networks. This allows for more control over networking and relieves some constraints of direct L2 connections. The primary technology used to stretch a VLAN across physical data centers is VxLAN, which allows for Layer 2 connectivity extension over a Layer 3 network, effectively creating a virtual overlay network that can span multiple data centers while maintaining logical segmentation within the VLANs. VxLAN is used by VMware's NSX-T offering, but other technologies also exist, such as L2TPv3 and MPLS. How can NSX-T minimize inter-site traffic? Because NSX-T can have logically distributed routers, we can define anoverlay-backed segment. In an overlay-backed segment, traffic between two VMs on different hosts but attached to the same overlay segment has their layer 2 traffic carried by a tunnel between the hosts. directly, bypassing the physical VLAN's. Diagram source In practice, this means that traffic between two VM's in the same segment - even if they are on different VLANs and different ESXi hosts - does not need to traverse the physical network gateway. I.e, if our VLAN's default gateway of ".1" exists on a physical router in Location A, but traffic is being sent between 2x VM's on different hosts and VLAN's in Location B, the traffic does not need to traverse the inter-site link. This is very powerful. To minimize VM-to-VM traffic crossing the inter-site link, we must configure NSX-T correctly, migrate all VM's in a given app/workload between data centers at the same time, and also at the same time move any F5 VIPs that process application traffic for that workload. Pre-migration BIG-IP overview The simplified diagram below shows the customer's pre-migration environment. The legacy datacenter still hosts almost all workloads, but the plumbing has been laid for migration. Using NSX-T, the VLANs in datacenter A are stretched to datacenter B. Existing BIG-IP's can reach pool members in Datacenter B. Any VM can be migrated between sites A and B without changing it's IP address. Post-migration BIG-IP end state The diagram below shows our post-migration goal. Let's remember our requirements. We want to get here with: HA maintained within each datacenter (we'll have 4 devices for a period of time) Minimal inter-site traffic (skip the physical VLAN gateway if possible) The easiest method possible (ie, sync configs and do not deploy new config on a disparate BIG-IP cluster) Migration options Let's review a few options for migration. 1. HA pair across physical locations Given that our end goal is to have all VM workloads and BIG-IP VE's in location B, we may be tempted to take a shortcut approach: just migrate one BIG-IP VE to site B and run an Active/Standby pair across datacenters. This could work in terms of connectivity, but raises some disadvantages for a production scenario: Latency. Only 1 BIG-IP can be Active. Latency will occur for all traffic between the Active BIG-IP and any nodes in the other datacenter. HA. Running a single HA pair across sites leaves us without High Availabilitywithineither site. Hard cutover. A cutover of the Active/Standy roles between site A to site B can be planned, but it's an "all or nothing" approach in terms of site A or B hosting the Active BIG-IP. There's no graceful way to keep both VM workloadsandthe Active BIG-IP in the same site and migrate together. I have personally managed HA pairs run across two physical datacenters with Layer 2 connectivity. In a scenario with very little latency between sites, or where a migration was not planned, that might be appropriate. However, in this case, the disadvantages listed here make this option less than ideal. 2. Second HA pair of BIG-IP devices in site B Given that our end goal is to have a single HA pair of devices in site B, we could build a brand new, disparate BIG-IP cluster in site B and migrate Virtual Servers from cluster A to B. After migration, decommission cluster A. This could work but raises unnecessary complications: IP conflicts. If we build 2x HA pairs, both on the same VLAN's, we have the potential for IP conflicts. We must migrate every Virtual Server and Pool by recreating our configuration on cluster B. This means every VIP must change. Tediousness. We could alleviate some pain by cleverly replicating the configuration from cluster A on cluster B, but disabling Virtual Addresses until a time of cutover. This would be possible but tedious, and introduce some risk. We could automate the deletion of VIP's and pools from one device, and creation on another, but this automation is likely to be difficult if the original configuration was not created with automation itself. DNS. If IP addresses of VirtualServers do change, we must consider the DNS changes that would be required. One advantage, however, is that two separate clusters is an architecture that is very easy to understand. 3. Single device cluster with 4x BIG-IPs This is my preferred approach when latency between sites is non-trivial and we must operate for some time with Active BIG-IPs in both locations. We'll temporarily grow our cluster from 2 devices to 4, with 2 devices in each cluster. We'll introduce an additional Traffic Group. Traffic Group 1 (existing TG) will be Active on BIG-IP 1 with BIG-IP 2 as next preferred. Traffic Group 2 (new TG) will be Active on BIG-IP 3 with BIG-IP 4 as next preferred. Pre-migration, all VIP's exist in TG 1. Migrate workloads components together: Migrate related VM's for workloads to Datacenter B. Migrate appropriate VIP's between TG 1 and TG 2. East-West traffic between workload VMs and BIG-IP should all remain within Location B. Once all VIP's are migrated to Traffic Group 2: Delete Traffic Group 1 Remove and decommission BIG-IP's 1 and 2. The advantages to his approach are: No IP address changes. Virtual Addresses of Virtual Servers do not change. Operationally smooth. A TMSH command can move a Virtual Address between Traffic Groups. Workloads can move app-by-app and not in an "all-or-nothing" approach. No DNS changes required. HA maintained within sites. Further details when configuring a single, 4-Device Cluster The basic idea behind this migration architecture is that a given VIP will be advertised via ARP fromeitherLocation A or B, depending on the Traffic Group to which the Virtual Address belongs. This allows us to have asingle Device Group (easiest because config is synced between all devices) and to usetwo Traffic Groups. The Device Group type is stillSync-Failover (and not Sync-Only). In a Sync-Only group, the /Common folder is not synced between member devices, but in our case the existing configuration exists within /Common and so we will need this replicated between all four devices. Multiple Device Groups within a single Trust Domain, as diagrammed in this example, are not planned in this scenario. Because partitions are mapped to a Device Group, and all of our existing configuration is within a single /Common partition, multiple Device Groups are not appropriate in this case. Failover is configured individually for each Traffic Group. TG 1 will failover between BIG-IP 1 & 2, and TG 2 will failover between BIG-IP 3 & 4. We often refer to two-node clusters as "Active/Standby" or "Active/Active". When a cluster has 3+ nodes we often refer to it as "ScaleN" or "Active/Active/Standby", or similar. For this scenario, we might use the term "Active/Standby/Active/Standby", since we'll have 2 Active devices and 4 devices total during our migration. Further questions for our customer scenario When do we migrate the default gateway for each of our physical VLANs? The physical VLAN's have a gateway - let's call it ".1" - currently configured on routers on Virginia. This is relevant because some traffic may traverse physical VLAN's: traffic between VM's that are actually running in different datacenters because they were not migrated together, traffic from regional sites that are routed by the WAN to Virginia, traffic to physical VM's that will be migrated separately, etc. Moving that ".1" from Virginia to Dallas will be once-off move that's best left to the customer's network team. Advanced network management technologies can distribute a VLAN and gateway for physical networks, just like our NSX-T example does for VLANs within the virtual environment. But this decision is outside of the recommendation for F5 BIG-IP migration. What about physical servers in Virginia? How do we handle those? Physical machines must be re-built in Location 2. Physical VLAN's can be stretched, just like VLAN's within the virtual environment. Or, physical machines may reside on VLAN's that are local to each datacenter. In the case of a stretched physical VLAN, a physical server that is a pool member in BIG-IP could maintain it's IP address and BIG-IP configuration would not change. If the IP address of a pool member does change, of course the BIG-IP configuration must be updated. This makes the migration of physical servers less automated than VM's. In this scenario, the BIG-IP's themselves are virtual machines. If they were physical appliances on a stretched VLAN, the same approach would apply for BIG-IP migration (single Device Groups with two Traffic Groups). What about self IP addresses? This is a potentially important detail. Each BIG-IP will have unique Self IP's in each VLAN to which it is connected. Let's walk through this from two perspectives: health monitoring and SNAT. Health monitoring of pool members will be sourced from the non-floating Self IP that is local to the Active BIG-IP. Health monitoring is not sourced from a floating Self IP, if one exists. By introducing two additional devices we will be adding two IP addresses from which pool members may receive health checks. In our customer's case, no firewall currently exists between BIG-IP and pool members, so it's very unlikely that health monitoring from a new IP address will break an application. But it is important to note: if an application is expecting health monitoring to come from one of the two existing BIG-IP Self IP's only, we may need to update that application. SNAT, which is extremely common, means connections between BIG-IP and pool members will be sourced from a Self IP. If a floating Self IP exists, SNAT connections will be sourced from this IP address, otherwise they will be sourced from the non-floating address. Floating Self IP's are assigned to a Traffic Group and will failover between devices accordingly. This means that when a VirtualAddress is updated from Traffic Group 1 to 2, SNAT'd connections will now be sourced from a new IP address. The same potential concern exists for SNAT as with health monitors: if applications or firewalls accept only pre-configured IP addresses, we'll need to update them. However with SNAT an administrator might plan ahead with SNAT Lists. SNAT Lists are also assigned to Traffic Groups, and an administrator might create a SNAT List dedicated for an application. This SNAT List could be migrated between Traffic Groups at the same time as a Virtual Address and associated VM workload migration. Be mindful if updating the Traffic Group of a floating self IP or SNAT List: if multiple applications are expecting specific source IPs for health monitors or SNAT'd connections, those applications may need to be migrated together. Conclusion A stretched VLAN allows additional options for VM migrations. Careful planning of F5 BIG-IP deployments, using multiple Traffic Groups, will allow for a smooth migration plan. Thanks for reading!BIG-IP Next: iRules pool routing
If you use routing in iRules with the pool command in BIG-IP and you’re starting to kick the tires on BIG-IP Next in your lab environments, please note the pool reference is not just the pool name. For example, in classic BIG-IP, if I had a pool named myPool then the command in my iRule would just bepool myPool. In BIG-IP Next (as of this publish date, they are working on restoring the relative path) you will need these additional details: Tenant Name Application Name Pool Service Name The format for the pool reference is then: pool /app/myTenantName:ApplicationName/pool/PoolServiceName Consider this partial AS3 declaration (stripped down to the necessary lines for brevity): { "class": "ADC", "tenant2zBLVQCbR2unEw5ge6nVrQ": { "class": "Tenant", "testapp1": { "class": "Application", "pool_reference_test_7_testvip1": { "class": "iRule", "iRule": { "base64": "" } }, "testpool1": { "class": "Pool" }, "testpool2": { "class": "Pool" }, "testpool2_service": { "class": "Service_Pool", "pool": "testpool2" }, "testpool3": { "class": "Pool" }, "testpool3_service": { "class": "Service_Pool", "pool": "testpool3" }, "testvip1": { "class": "Service_HTTP", "iRules": [ "pool_reference_test_7_testvip1" ], "pool": "testpool1", "virtualAddresses": [ "10.0.2.51" ], "virtualPort": 80 } } } } In this case, there is a default pool (testpool1) attached to the virtual server, but the ones that will require routing in the iRule, testpool2 and testpool3, are not attached. They are mapped in the Service_Pool classes though, and that's what we need for the iRule. From this declaration, we need: Tenant Name: tenant2zBLVQCbR2unEw5ge6nVrQ Application Name: testapp1 Service Pool Names: testpool2_service, testpool3_service The original iRule then, as shown here: when HTTP_REQUEST { if { [string tolower [HTTP::uri]] == "/tp2" } { pool testpool2 HTTP::uri / } elseif { [string tolower [HTTP::uri]] == "/tp3" } { pool testpool3 HTTP::uri / } } Becomes: when HTTP_REQUEST { if { [string tolower [HTTP::uri]] == "/tp2" } { pool /app/tenant2zBLVQCbR2unEw5ge6nVrQ:testapp1/pool/testpool2_service HTTP::uri / } elseif { [string tolower [HTTP::uri]] == "/tp3" } { pool /app/tenant2zBLVQCbR2unEw5ge6nVrQ:testapp1/pool/testpool3_service HTTP::uri / } } When creating an application service in the Central Manager GUI, here’s the workflow I used: Create the application service without the iRule, but with whatever pools you’re going to route to so that the pools and pool services are defined. Validate the app and view results. This is where you’ll find your tenant and service pool names. The app’s name should be obvious as you set it! Go ahead and deploy; there isn’t a way here to save drafts currently. Create or edit the iRule with the pool format above with your details. Edit the deployment to reference your iRule (and the correct version), then redeploy. This should get you where you need to be! Comment below or start a thread in the forums if you get stuck.How I did it - "Securing NVIDIA’s Morpheus AI Framework with NGINX Plus Ingress Controller”
In this installment of "How I Did It," we continue our journey into AI security. I have documented how I deployed an NVIDIA Morpheus AI infrastructure along with F5's NGINX Plus Ingress Controller to provide secure and scalable external access.Service Discovery in F5 Distributed Cloud - Architecture Options
The F5 Distributed Cloud (XC) Virtual Edition (VE) Customer Edge (CE) platform can be deployed within your data center or cloud environment. It can perform service discovery for services in your Kubernetes (K8s) clusters. Why do Service discovery? Service discovery is important in systems that change and move around, like microservices architectures. It helps find and connect services automatically. Instead of hard coding network locations, service discovery makes sure that services can easily find and communicate with each other, even when they scale or change locations. This improves scalability, resilience, and simplifies managing services in complex environments like Kubernetes or cloud infrastructures. By reducing manual intervention, service discovery enhances the overall efficiency and reliability of application deployments. The F5 Distributed Cloud (XC) CE can use the native kube-apiserver, or Consul, to query for services as they come online enabling admins to reference these discovered services. These services become XC origin pool definitions and can then be published locally through a proxy (http load balancer) on the CE itself or via our Global Application Delivery Network (ADN) - (Regional Edge Deployment). The F5 XC Load Balancer does more than just balance packets. It offers a set of SAAS security services that are easy to use. Customers can have a globally redundant layer of security while serving content from private K8’s clusters. This write-up covers two distinct service discovery architecture options available with XC. Secure K8s Gateway (VE CE) Kubernetes Sitetype Customer Edge (K8s sitetype CE) Depending on your service discovery use-case you may end up with one of these two options or both as they are not mutually exclusive. The first option of using the CE as a Secure K8s Gateway, may be the easier option for folks not particularly versed with the nuances of Kubernetes. Architecture 1:Virtual Edition Customer Edge (VE CE) If a picture is worth a thousand words then a working lab environment is worth a million. This repo walks through an entire Secure K8s GW setup and will leave you with a config that could easily be expanded upon. You can quickly build a PoC and start getting familiar with these modern app capabilities by using these tools. The readme includes details on how to use everything and what functions the various tools provide. It's all shell script and yaml so it's very easy to read through these and understand what's going on. https://github.com/dober-man/ve-ce-secure-k8s-gw This repo is designed to automate the deployment and configuration of a secure Kubernetes Gateway in the F5 Distributed Cloud (F5 XC) environment. It provides scripts and YAML configurations to set up secure communication, networking policies, and infrastructure components required to establish a secure gateway in a Kubernetes cluster. The readme file also documents the pre-reqs but you will at a minimum need an XC tenant, an XC Virtual Edition CE and an Ubuntu 22.04 server. If you do not have an XC tenant or VE CE, reach out to your local F5 Account team. Please use the issues feature of Github to report any discrepancies with the builds or documentation. Architecture 2:Kubernetes Sitetype Customer Edge (K8s sitetype CE) In this architecture, the entire CE runs as a service within the cluster. This model is going to require a bit more fundamental knowledge of Kubernetes and more tightly integrates with the cluster. You can quickly build a PoC and start getting familiar with these modern app capabilities by using this repo. https://github.com/dober-man/k8s-sitetype-ce This repo is focused on automating the deployment of the k8s-sitetype CE in a Kubernetes cluster. It provides scripts to simplify the process of setting up a secure site gateway for handling network traffic between cloud environments, on-premises infrastructure, and edge locations. The readme file documents the pre-reqs, but you will at a minimum need an XC tenant and an Ubuntu 22.04 server. If you do not have an XC tenant, reach out to your local F5 Account team. Please use the issues feature of Github to report any discrepancies with the builds or documentation. Summary - F5 Distributed Cloud offers a number of Kubernetes integration options for service discovery but also offers several other capabilities including Virtual K8s (Namespace as a Service) and Managed K8s which will be covered in future articles. Please feel free to drop a like or leave a comment below.OWASP Tactical Access Defense Series: Broken Object Level Authorization and BIG-IP APM
Addressing Broken Object Level Authorization (BOLA) vulnerabilities requires a multifaceted approach, combining robust coding practices, secure development methodologies, and powerful tools. Among these tools, F5 BIG-IP Access Policy Manager (APM) stands out as a crucial component in the arsenal of security measures. This article, the second in a series of articles dedicated to fortifying application security, delves into the pivotal role that BIG-IP APM plays in identifying, mitigating, and ultimately preventing OWASP top 10 API vulnerabilities byproviding developers and security professionals with a comprehensive guide to bolstering application security in the face of evolving cyber threats. Broken Object Level Authorization This is one of the most common and severe vulnerabilities within APIs and is related to Insecure Direct Object References (IDOR). Starting with, what's Object Level Authorization? This is an access control mechanism that's in place to validate which user has access to a specific endpoint and what actions to be performed. BOLA and IDOR refer to situations where the endpoints fail to enforce specific authorization rules on endpoints, or the user is successfully able to access unauthorized endpoints and perform unauthorized actions. The weakness that can lead to this vulnerability is the server component fails to track client state and rely on other parameters that can be tweaked from the client side, for example (Cookies, object IDs). BOLA Example Let's assume this backend directory, - /uploads/ - user1/ - file1.txt - file2.txt - user2/ - file3.txt - file4.txt The expected user1 usage is as follows, https://example.com/viewfile?file=file1.txt the user can access file1. If the server is vulnerable to BOLA, let's have user2 accessing the server, then try to navigate to file1 as follows, https://example.com/viewfile?file=user1/file1.txt What could help us in this situation? Yes, we need granular endpoint authorization with proper client state tracking. That's where our lovely friend BIG-IP APM comes into the picture. Let's see how BIG-IP APM can help us. BIG-IP APM and BOLA protection BIG-IP APM provides API protection through its Per-Request policy, where the it applies granular Access protection to each API endpoint. How BIG-IP APM enhancesdefenses We start with creating our Per-Request policy, this policy works in a different way than the per-session policy, as the flow will be evaluted on a per-request basis, making sure to consider variations throught the session life-time. Below are some of the key benefits: Wide range of Authentication, SSO, and MFA mechanisms to properly identify the initiating machine or user. Ability to integrate with 3rd parties to provide additional enforcement decisions based on the organization's policy. Ability to apply endpoint checks on the client side before session initiation. This goes to BIG-IP in general, the ability to apply custom traffic control on both of the traffic sides, Client and Server. Using BIG-IP API protection profile. Protection profiles are an easy way to deploy both APM (Per-Request policy) and Advanced Web Application Firewall (AWAF). As a pre-requisite, you need APM, AWAF licensed and provisioned. Use OpenAPI Spec 2.0 as an input to the API protection. Apply different Authentication methods, whether Basic, Oauth (Directly from the template), or once we have the API protection profile created, we can customize policy elements to our needs. Using Hybrid approach with F5 Distributed Cloud (F5 XC) + BIG-IP APM We had this approach discussed in details through F5Hybrid Security Architectures (Part 5 - F5 XC, BIG-IP APM, CIS, and NGINX Ingress Controller) Stay engaged to explore the comprehensive capabilities of BIG-IP APM and how it plays a pivotal role in fortifying your security posture against these formidable threats. Related Content F5 BIG-IP Access Policy Manager | F5 Introduction to OWASP API Security Top 10 2023 OWASP Top 10 API Security Risks – 2023 - OWASP API Security Top 10 API Protection Concepts OWASP Tactical Access Defense Series: How BIG-IP APM Strengthens Defenses Against OWASP Top 10 F5 Hybrid Security Architectures (Part 5 - F5 XC, BIG-IP APM, CIS, and NGINX Ingress Controller)Experience the power of F5 NGINX One with feature demos
Introduction Introducing F5 NGINX One, a comprehensive solution designed to enhance business operations significantly through improved reliability and performance. At the core of NGINX One is our data plane, which is built on our world-class, lightweight, and high-performance NGINX software. This foundation provides robust traffic management solutions that are essential for modern digital businesses. These solutions include API Gateway, Content Caching, Load Balancing, and Policy Enforcement. NGINX One includes a user-friendly, SaaS-based NGINX One Console that provides essential telemetry and overseas operations without requiring custom development or infrastructure changes. This visibility empowers teams to promptly address customer experience, security vulnerabilities, network performance, and compliance concerns. NGINX One's deployment across various environments empowers businesses to enhance their operations with improved reliability and performance. It is a versatile tool for strengthening operational efficiency, security posture, and overall digital experience. NGINX One has several promising features on the horizon. Let's highlight three key features: Monitor Certificates and CVEs, Editing and Update Configurations, and Config Sync Groups. Let's delve into these in details. Monitor Certificates and CVE’s: One of NGINX One's standout features is its ability to monitor Common Vulnerabilities and Exposures (CVEs) and Certificate status. This functionality is crucial for maintaining application security integrity in a continually evolving threat landscape. The CVE and Certificate Monitoring capability of NGINX One enables teams to: Prioritize Remediation Efforts: With an accurate and up-to-date database of CVEs and a comprehensive certificate monitoring system, NGINX One assists teams in prioritizing vulnerabilities and certificate issues according to their severity, guaranteeing that essential security concerns are addressed without delay. Maintain Compliance: Continuous monitoring for CVEs and certificates ensures that applications comply with security standards and regulations, crucial for industries subject to stringent compliance mandates. Edit and Update Configurations: This feature empowers users to efficiently edit configurations and perform updates directly within the NGINX One Console interface. With Configuration Editing, you can: Make Configuration Changes: Quickly adapt to changing application demands by modifying configurations, ensuring optimal performance and security. Simplify Management: Eliminate the need to SSH directly into each instance to edit or update configurations. Reduce Errors: The intuitive interface minimizes potential errors in configuration changes, enhancing reliability by offering helpful recommendations. Enhance Automation with NGINX One SaaS Console: Integrates seamlessly into CI/CD and GitOps workflows, including GitHub, through a comprehensive set of APIs. Config Sync Groups: The Config Sync Group feature is invaluable for environments running multiple NGINX instances. This feature ensures consistent configurations across all instances, enhancing application reliability and reducing administrative overhead. The Config Sync Group capability offers: Automated Synchronization: Configurations are seamlessly synchronized across NGINX instances, guaranteeing that all applications operate with the most current and secure settings. When a configuration sync group already has a defined configuration, it will be automatically pushed to instances as they join. Scalability Support: Organizations can easily incorporate new NGINX instances without compromising configuration integrity as their infrastructure expands. Minimized Configuration Drift: This feature is crucial for maintaining consistency across environments and preventing potential application errors or vulnerabilities from configuration discrepancies. Conclusion NGINX One Cloud Console redefines digital monitoring and management by combining all the NGINX core capabilities and use cases. This all-encompassing platform is equipped with sophisticated features to simplify user interaction, drastically cut operational overhead and expenses, bolster security protocols, and broaden operational adaptability. Read our announcement blog for moredetails on the launch. To explore the platform’s capabilities and see it in action, we invite you to tune in to our webinar on September 25th. This is a great opportunity to witness firsthand how NGINX One can revolutionize your digital monitoring and management strategies.Ridiculously Easy Bot Protection: How to Use BIG-IP APM to Streamline Bot Defense Implementation
Ever imagined how your Bot solution implementation would be with a standard entry page at your application side--a page that’s easily referred, with clear parameters, and structured customization options? In this article, we are exploring using F5 BIG-IP Access Policy Manager (BIG-IP APM) along side F5 Distributed Cloud Bot Defense (XC Bot Defense). Bot defense solutions' challenges Implementing bot defense solutions presents several challenges, each with unique considerations: Evolving Bot Tactics: Bot tactics constantly evolve, demanding adaptive detection methods to avoid both false positives (blocking legitimate users) and false negatives (allowing malicious bots through). Effective solutions must be highly flexible and responsive to these changes. Multi-Environment Integration: Bot defenses need to be deployed across diverse environments, including web, mobile, and APIs, adding layers of complexity to integration. Ensuring seamless protection across these platforms is critical. Balancing Security and Performance: Security measures must be balanced with performance to avoid degrading the user experience. A well-calibrated bot defense should secure the application without causing noticeable slowdowns or other disruptions for legitimate users. Data Privacy Compliance: Bot solutions often require extensive data collection, so adherence to data privacy laws is essential. Ensuring that bot defense practices align with regulatory standards helps avoid legal complications and maintains user trust. Resource Demands: Integrating bot defense with existing security stacks can be resource-intensive, both in terms of cost and skilled personnel. Proper configuration, monitoring, and maintenance require dedicated resources to ensure long-term effectiveness and efficiency. What F5 BIG-IP APM brings to the table? For teams working on bot defense solutions, several operational challenges can arise: Targeted Implementation Complexity: Identifying the correct application page for applying bot defense is often a complex process. Teams must ensure the solution targets the page containing the specific parameters they want to protect, which can be time-consuming and resource-intensive. Adaptation to Application Changes: Changes like upgrades or redesigns of the application page often require adjustments to bot defenses. These modifications can translate into significant resource commitments, as teams work to ensure the bot solution remains aligned with the new page structure. BIG-IP APM simplifies this process by making it easier to identify and target the correct page, reducing the time and resources needed for implementation. This allows technical and business resources to focus on more strategic priorities, such as fine-tuning bot defenses, optimizing protection, and enhancing application performance. Architecture and traffic flow In this section, let's explore how F5 XC Bot Defense and BIG-IP APM works together, let's list the prerequisites, F5 XC account with access to Bot Defense. APM licensed and provisioned. F5 BIG-IP min. v16.x for native connector integration. BIG-IP Self IP rechability to Internet to communicate with F5 XC, mainly to reach this domin (ibd-web.fastcache.net). Now, time to go quickly through our beloved TMM packet order. Due to the nature of BIG-IP APM Access events take precedence to the Bot enforcement, hence we will rely on simple iRule to apply Bot Defense on BIG-IP APM logon page. BIG-IP Bot Defense is responsible for inserting the JS and passing traffic from client to APM VS back and forth. BIG-IP APM responsible for logon page, MFA, API security or SSO integrations to manage client Access to the backend application. Solution Implementation Let's start now with our solution implementation, F5 Distributed Cloud Bot defense connector with BIG-IP was discussed in details in this Article F5 Distributed Cloud Bot Defense on BIG-IP 17.1 You will follow the steps mentioned in the article, with few changes mentioned below, API Hostname Web: ibd-web.fastcache.net For Per-session policies we use /my.policy as the target URL, while for Per-request and MFA implementation, you need to add /vdesk/*. Protection Pool - Web: Create pool with FQDN ibd-web.fastcache.net Virtual server, Create LTM virtual server to listen to incoming traffic, perform SSL offloading, HTTP profile and attach Bot Defense connector profile. Forwarding iRule, attach forwarding irule to the Bot virtual server. when CLIENT_ACCEPTED { ## Forwarding to the APM Virtual Server virtual Auth_VS } BIG-IP APM Policies,In this step we are creating two options of our deployment, Per-Session policy, where BIG-IP presents Logon page to the user. Per-Request policy, which services in case initial logon is handled at remote IdP and APM handles Per-request, MFA authentication or API security. Now, it's time to run the traffic and observe the results, From client browser, we can see the customer1.js inserted, From F5 XC Dashboard, Conclusion The primary goal of incorporating BIG-IP APM into the Bot Defense solution is to strike a balance between accelerating application development across web and mobile platforms while enforcing consistent organizational bot policies. By decoupling application login and authentication from the application itself, this approach enables a more streamlined, optimized, and secure bot defense implementation. It allows development teams to concentrate on application performance and feature enhancements, knowing that security measures are robustly managed and seamlessly integrated into the infrastructure. Related Content F5 Distributed Cloud Bot Defense on BIG-IP 17.1 Bot Detection and Security: Stop Automated Attacks 2024 Bad Bots ReviewF5 NGINX Plus R33 Licensing and Usage Reporting
Beginning with F5 NGINX Plus version R33, all customers are required to deploy a JSON Web Token (JWT) license for each commercial instance of NGINX Plus. Each instance is responsible for validating its own license status. Furthermore, NGINX Plus will report usage either to the F5 NGINX licensing endpoint or to the F5 NGINX Instance Manager for customers who are connected. For those customers who are disconnected or operate in an air-gapped environment, usage can be reported directly to the F5 NGINX Instance Manager. To learn more about the latest features of NGINX R33, please check out the recent blog post. Install or Upgrade NGINX Plus R33 To successfully upgrade to NGINX Plus R33 or perform a fresh installation, begin by downloading the JWT license from your F5 account. Once you have the license, place it in the F5 NGINX directory before proceeding with the upgrade. For a fresh installation, after completing the installation, also place the JWT license in the NGINX directory. For further details, please refer to the provided instructions. This video provides a step-by-step guide on installing or upgrading to NGINX Plus R33. Report Usage to F5 in Connected Environment To effectively report usage data to F5 within a connected environment using NGINX Instance Manager, it's important to ensure that port 443 is open. The default configuration directs the usage endpoint to send reports directly to the F5 licensing endpoint at product.connect.nginx.com. By default, usage reporting is enabled, and it's crucial to successfully send at least one report on installation for NGINX to process traffic. However, you can postpone the initial reporting requirement by turning off the directive in your NGINX configuration. This allows NGINX Plus to handle traffic without immediate reporting during a designated grace period. To configure usage reporting to F5 using NGINX Instance Manager, update the usage endpoint to reflect the fully qualified domain name (FQDN) of the NGINX Instance Manager. For further details, please refer to the provided instructions. This video shows how to report usage in the connected environment using NGINX Instance Manager. Report Usage to F5 in Disconnected Environment using NGINX Instance Manager In a disconnected environment without an internet connection, you need to take certain steps before submitting usage data to F5. First, in NGINX Plus R33, update the `usage report` directive within the management block of your NGINX configuration to point to your NGINX Instance Manager host. Ensure that your NGINX R33 instances can access the NGINX Instance Manager by setting up the necessary DNS entries. Next, in the NMS configuration in NGINX Instance Manager, modify the ‘mode of operation’ to disconnected, save the file, and restart NGINX Instance Manager. There are multiple methods available for adding a license and submitting the initial usage report in this disconnected environment. You can use a Bash script, REST API, or the web interface. For detailed instructions on each method, please refer to the documentation. This video shows how to report usage in disconnected environments using NGINX Instance Manager. Conclusion The transition to NGINX Plus R33 introduces important enhancements in licensing and usage reporting that can greatly improve your management of NGINX instances. With the implementation of JSON Web Tokens (JWT), you can validate your subscription and report telemetry data more effectively. To ensure compliance and optimize performance, it’s crucial to understand the best practices for usage reporting, regardless of whether you are operating in a connected or disconnected environment. Get started today with a 30-day trial, and contact us if you have any questions. Resources NGINX support documentation Blog announcementproviding a comprehensive summary of the new features in this release.
Use Fully Qualified Domain Name (FQDN) for GSLB Pool Member with F5 DNS
Normally, we define a specific IP (and port) to be used as GSLB pool member. This article provides a custom configuration to be able to use Fully Qualified Domain Name (FQDN) as GSLB pool member--with all GSLB features like health-check monitoring, load balancing method, persistence, etc. Despite GSLB as a mechanism to distribute traffic across datacenters having reached years of age, it has not become less relevant this recent years. The fact that internet infrastructure still rely heavily on DNS technology means GSLB is continuously used due to is lightweight nature and smooth integration. When using F5 DNS as GSLB solution, usually we are dealing with LTM and its VS as GSLB server and pool member respectively. Sometimes, we will add a non-LTM node as a generic server to provide inter-DC load balancing capability. Either way, we will end up with a pair of IP and port to represent the application, in which we sent a health-check against. Due to the trend of public cloud and CDN, there is a need to use FQDN as GSLB pool member (instead of IP and port pair). Some of us may immediately think of using a CNAME-type GSLB pool to accommodate this. However, there is a limitation in which BIG-IP requires a CNAME-type GSLB pool to use a wideIP-type pool member, in which we will end up with an IP and port pair (again!) We can use "static target", but there is "side-effect" where the pool member will always consider available (which then triggers the question why we need to use GSLB in the first place!). Additionally, F5 BIG-IP TMUI accepts FQDN input when we configure GSLB server and pool member. However, it will immediately translate to IP based on configured DNS. Thus, this is not the solution we are looking for Now this is where F5’s BIG-IP power (a.k.a programmability) comes into play. Enter the realm of customization... We all love customization, but at the same time do not want that to be overly complicated so that life becomes harder on day-2 🙃. Thus, the key is to use some customization, but simple enough to avoid unnecessary complication. Here is one idea to solve our FQDN as GSLB pool problem above The customized configuration object includes 1. External health-check monitor: Dynamically resolve DNS to translate FQDN into IP address Perform health-check monitoring against current IP address Result is used to determine GSLB pool member availability status 2. DNS iRules: Check #1: Checks if GSLB pool attached to wideIP contains only FQDN-type member (e.g. other pool referring to LTM VS is also attached to the wideIP) If false, do nothing (let DNS response refer to LTM VS) Otherwise, perform check #2 Check #2: Checks current health-check status of requested domain name If FQDN is up, modify DNS response to return current IP of FQDN Otherwise, perform fallback action as requirement (e.g. return empty response, return static IP, use fallback pool, etc.) 3. Internal Datagroup: Store current IP of FQDN, updated according to health-check interval Datagroup record value contains current IP if health-check success. Otherwise, the value contains empty data Here are some of the codes, where configured; wideIP is gslb.test.com, while GSLB pool member FQDN is arcadia.f5poc.id 1. External health-check monitor config gtm monitor external gslb_external_monitor { defaults-from external destination *:* interval 10 probe-timeout 5 run /Common/gslb_external_monitor_script timeout 120 #define FQDN here user-defined fqdn arcadia.f5poc.id } External health-check monitor script #!/bin/sh pidfile="/var/run/$MONITOR_NAME.$1..$2.pid" if [ -f $pidfile ] then kill -9 -`cat $pidfile` > /dev/null 2>&1 fi echo "$$" > $pidfile # Obtain current IP for the FQDN resolv=`dig +short ${fqdn}` # The actual monitoring action here curl -fIs -k https://${fqdn}/ --resolve ${fqdn}:443:${resolv} | grep -i HTTP 2>&1 > /dev/null status=$? if [ $status -eq 0 ] then # Actions when health-check success rm -f $pidfile tmsh modify ltm data-group internal fqdn { records replace-all-with { $fqdn { data $resolv } } } echo "sending monitor to ${fqdn} ${resolv} with result OK" | logger -p local0.info echo "up" else # Actions when health-check fails tmsh modify ltm data-group internal fqdn { records replace-all-with { $fqdn { } } } echo "sending monitor to ${fqdn} ${resolv} with result NOK" | logger -p local0.info fi rm -f $pidfile 2. DNS iRules when DNS_REQUEST { set qname [DNS::question name] # Obtain current IP for the FQDN set currentip [class match -value $qname equals fqdn] } when DNS_RESPONSE { set rname [getfield [lindex [split [DNS::answer]] 4] "\}" 1 ] #Check if return is IP address of specially encoded FQDN IP, 10.10.10.10 in this example if {$rname eq "10.10.10.10" }{ #Response is only from pool with external monitor, meaning no other pool is attached to wideIP if {$currentip ne ""}{ #Current FQDN health-check success DNS::answer clear # Use current IP to construct DNS answer section DNS::answer insert "[DNS::question name]. 123 [DNS::question class] [DNS::question type] $currentip" } else { #Current FQDN health-check failed #Define action to be performed here DNS::answer clear } } } 3. Internal Datagroup ltm data-group internal fqdn { records { # Define FQDN as record name arcadia.f5poc.id { # Record data contains IP, where this will be continuously updated by external monitoring script data 158.140.176.219 } } type string } *GSLB virtual server configuration Some testing The resolve will follow whichever current IP address for the FQDN. If a returning CNAME response is required, you can do so by modifying DNS irules above. The logic and code are open to any improvement, so leave your suggestions in the comments if you have any. Thanks!
