Using Distributed Cloud DNS Load Balancer with Geo-Proximity and failover scenarios
Introduction To have both high performance and responsive apps available on the Internet, you need a cloud DNS that’s both scalable and one that operates at a global level to effectively connect users to the nearest point of presence. The F5 Distributed Cloud DNS Load Balancer positions the best features used with GSLB DNS to enable the delivery of hybrid and multi-cloud applications with compute positioned right at the edge, closest to users. With Global Server Load Balancing (GSLB) features available in a cloud-based SaaS format, the Distributed Cloud DNS Load Balancer has a number distinct advantages: Speed and simplicity: Integrate with DevOps pipelines, with an automation focus and a rich and intuitive user interface Flexibility and scale: Global auto-scale keeps up with demand as the number of apps increases and traffic patterns change Security: Built-in DDoS protection, automatic failover, and DNSSEC features help ensure your apps are effectively protected. Disaster recovery: With automatic detection of site failures, apps dynamically fail over to individual recovery-designated locations without intervention. Adding user-location proximity policies to DNS load balancing rules allows the steering of users to specific instances of an app. This not only improves the overall experience but it guarantees and safeguards data, effectively silo’ing user data keeping it region-specific. In the case of disaster recovery, catch-all rules can be created to send users to alternate destinations where restrictions to data don’t apply. Integrated Solution This solution uses a cloud-based Distributed Cloud DNS to load balance traffic to VIP’s that connect to region-specific pools of servers. When data privacy isn’t a requirement, catch-all rules can further distribute traffic should a preferred pool of origin servers become unhealthy or unreachable. The following solution covers the following three DNS LB scenarios: Geo-IP Proximity Active/Standby failover within a region Disaster Recovery for manually activated failovers Autonomous System Number (ASN) Lists Fallback pool for automated failovers The configuration for this solution assumes the following: The app is in multiple regions Users are from different regions Distributed Cloud hosts/manages/is delegated the DNS domain or subdomain (optional) Failover to another region is allowed Prerequisite Steps Distributed Cloud must be providing primary DNS for the domain. Your domain must be registered with a public domain name registrar with the nameservers ns1.f5clouddns.com and ns2.f5clouddns.com. F5 XC automatically validates the domain registration when configured to be the primary nameserver. Navigate to DNS Management > domain > Manage Configuration > Edit Configuration >> DNS Zone Configuration: Primary DNZ Configuration > Edit Configuration. Select “Add Item”, with Record Set type “DNS Load Balancer” Enter the Record Name and then select Add Item to create a new load balancer record. This opens the submenu to create DNS Load Balancer rules. DNS LB for Geo-Proximity Name the rule “app-dns-rule” then go to Load Balancing Rules > Configure. Select “Add Item” then under the Load Balancing Rule, within the default Geo Location Selection, expand the “Selector Expression” and select “geoip.ves.io/continent”. Select Operator “In” and then the value “EU”. Click Apply. Under the Action “Use DNS Load Balancer pool”, click “Add Item”. Name the pool “eu-pool”, and under Pool Type (A) > Pool Members, click “Add Item”. Enter a “Public IP”, then click “Apply”. Repeat this process to have a second IP Endpoint in the pool. Scroll down to Load Balancing Method and select “Static-Persist”. Now click Continue, and then Apply to the Load Balancing Rule, and then “Add Item” to add a second rule. In the new rule, choose Geo Location Selection value “Geo Location Set selector”, and use the default “system/global-users”. Click “Add Item”. Name this new pool “global-pool” and add then select “Add Item” with the following pool member: 54.208.44.177. Change the Load Balancing Mode to “Static-Persist”, then click Continue. Click “Continue”. Now set the Load Balancing Rule Score to 90. This allows the first load balancing rule, specific to EU users, to be returned as the only answer for users of that region unless the regional servers are unhealthy. Note: The rule with the highest score is returned. When two or more rules match and have the same score, answers for each rule is returned. Although there are legitimate reasons for doing this, matching more than one rule with the same score may provide an unanticipated outcome. Now click "Apply", “Apply”, and “Continue”. Click the final “Apply” to create the new DNS Zone Resource Record Set. Now click “Apply” to the DNS Zone configuration to commit the new Resource Record. Click “Save and Exit” to finalize everything and complete the DNS Zone configuration! To view the status of the services that were just created, navigate to DNS Management > Overview > DNS Load Balancers > app-dns-rule. Clicking on the rule “eu-pool”, you can find the status for each individual IP endpoint, showing the overall health of each pool’s service that has been configured. With the DNS Load Balancing rule configured to connect two separate regions, when one of the primary sites goes down in the eu-pool users will instead be directed to the global-pool. This provides reliability in the context of site failover that spans regions. If data privacy is also a requirement, additional rules can be configured to support more sites in the same region. DNS LB for Active-Passive Sites In the previous scenario, two members are configured to be equally active for a single location. We can change the weight of the pool members so that of the two only one is used when the other is unhealthy or disabled. This creates a backup/passive scenario within a region. Navigate to DNS Load Balancer Management > DNS Load Balancers. Go to the service name "app-dns-rule", then under Actions, select Manage Configuration. Click Edit Configuration for the DNS rule. Go to the Load Balancing Rules section, and Edit Configuration. On the Load Balancing Rules order menu, go to Actions > Edit for the eu-pool Rule Action. In the Load Balancing Rule menu for eu-pool, under the section Action, click Edit Configuration. In the rule for eu-pool, under Pool Type (A) > Pool Members click the Edit action In the IP Endpoint section, change the Load Balancing Priority to 1, then click Apply. Change the Load Balancing Mode to Priority, then exit and save all changes by clicking Continue, Apply, Apply, and then Save and Exit. DNS LB for Disaster Recovery Unlike with backup/standby where failover can happen automatically depending on the status of a service's health, disaster recovery (DR) can either happen automatically or be configured to require manual intervention. In the following two scenarios, I'll show how to configure manual DR failover within a region, and also how to manual failover outside the region. To support east/west manual DR failover within the EU region, use the steps above to create a new Load Balancing Rule with the same label selector as the EU rule (eu-pool) above, then create a new DNS LB pool (name it something like eu-dr-pool) and add new designated DR IP pool endpoints. Change the DR Load Balancing Rule Score to 80, and then click Apply. On the Load Balanacing Rules page, change the order of the rules and confirm that the score is such that it aligns to the following image, then click Apply, and then Save and Exit. In the previous active/standby scenario the Global rule functions as a backup for EU users when all sites in EU are down. To force a non-regional failover, you can change F5 XC DNS to send all EU users to the Global DNS rule by disabling each of the two EU DNS rule(s) above. To disable the EU DNS rules, Navigate to DNS Load Balancer Management > DNS Load Balancers, and then under Actions, select Manage Configuration. Click Edit Configuration for the DNS rule. Go to the Load Balancing Rules section, and Edit Configuration. On the Load Balancing Rules order menu, go to Actions > Edit for the eu-pool Rule Action. In the Load Balance Rule menu for eu-pool, under the section Action, click Edit Configuration. In the top section labeled Metadata, check the box to Disable the rule. Then click Continue, Apply, Apply, and then Save and Exit. With the EU DNS LB rules disabled, all requests in the EU regionare served by the Global Pool. When it's time to restore regional services, all that's needed is to re-enter the configuration rule and uncheck the Disable box to each rule. DNS LB with ASN Lists ASN stands for Autonomous System Number. It is a unique identifier assigned to networks on the internet that operate under a single administration or entity. By mapping IP addresses to their corresponding ASN, DNS LB administrators can manage some traffic more effectively. To configure Distributed Cloud DNS LB to use ASN lists, navigate toDNS Management > DNS Load Balancer Management, then "Managed Configuration" for a DNS LB service. Choose "Add Item", and on the next page, select "ASN List", and enter one or more ASN's that apply to this rule, select a DNS LB pool, and optionally configure the score (weight). When the same ASN exists in multiple DNS LB rules, the rule having the highest score is used. Note: F5 XC uses ASPlain (4-byte) formatted AS numbers. Multiple numbers are configured one per item line. DNS LB Fallback Pool (Failsafe) The scenarios above illustrate how to designate alternate pools both regional and global when an individual pool fails. However, in the event of a catastrophic failure that brings all service pools are down, F5 XC provides one final mechanism, the fallback pool. Ideally, when implemented, the fallback pool should be independent from all existing pool-related infrastructure and services to deliver a failsafe service. To configure the Fallback Pool, navigate to DNS Management > DNS Load Balancer Management, then "Managed Configuration" of your DNS LB service. Click "Edit Configuration", navigate to the "Fallback Pool" box and choose an existing pool. If no qualified pool exists, the option is available to add a new pool. In my case, I've desginated "global-poolx" as my failsafe fallback pool which already functions as a regional backup. Best practice for the fallback pool is that it should be a pool not referenced elsewhere in the DNSLB configuration, a pool that exists on completely independent resources not regionally-bound. DNS LB Health Checks and Observability For sake of simplicity the above scenarios do not have DNS LB health checks configured and it's assumed that each pool's IP members are always reachable and healthy. My next article shows how to configure health checks to enable automatic failovers and ensure that users always reach a working server. Conclusion Using the Distributed Cloud DNS Load Balancer enables better performance of your apps while also providing greater uptime. With scaling and security automatically built into the service, responding to large volumes of queries without manual intervention is seamless. Layers of security deliver protection and automatic failover. Built-in DDoS protection, DNSSEC, and more make the Distributed Cloud DNS Load Balancer an ideal do-it-all GSLB distributor for multi-cloud and hybrid apps. To see a walkthrough where I configure first scenario above for Geo-IP proximity, watch the following accompanying video. Additional Resources Next article: Using Distributed Cloud DNS Load Balancer health checks and DNS observability More information about Distributed Cloud DNS Load Balancer available at: https://www.f5.com/cloud/products/dns-load-balancer Product Documentation: DNS LB Product Documentation DNS Zone ManagementUsing BIG-IP GTM to Integrate with Amazon Web Services
This is the latest in a series of DNS articles that I've been writing over the past couple of months. This article is taken from a fantastic solution that Joe Cassidy developed. So, thanks to Joe for developing this solution, and thanks for the opportunity to write about it here on DevCentral. As a quick reminder, my first six articles are: Let's Talk DNS on DevCentral DNS The F5 Way: A Paradigm Shift DNS Express and Zone Transfers The BIG-IP GTM: Configuring DNSSEC DNS on the BIG-IP: IPv6 to IPv4 Translation DNS Caching The Scenario Let's say you are an F5 customer who has external GTMs and LTMs in your environment, but you are not leveraging them for your main website (example.com). Your website is a zone sitting on your windows DNS servers in your DMZ that round robin load balance to some backend webservers. You've heard all about the benefits of the cloud (and rightfully so), and you want to move your web content to the Amazon Cloud. Nice choice! As you were making the move to Amazon, you were given instructions by Amazon to just CNAME your domain to two unique Amazon Elastic Load Balanced (ELB) domains. Amazon’s requests were not feasible for a few reasons...one of which is that it breaks the RFC. So, you engage in a series of architecture meetings to figure all this stuff out. Amazon told your Active Directory/DNS team to CNAME www.example.com and example.com to two AWS clusters: us-east.elb.amazonaws.com and us-west.elb.amazonaws.com. You couldn't use Microsoft DNS to perform a basic CNAME of these records because of the BIND limitation of CNAME'ing a single A record to multiple aliases. Additionally, you couldn't point to IPs because Amazon said they will be using dynamic IPs for your platform. So, what to do, right? The Solution The good news is that you can use the functionality and flexibility of your F5 technology to easily solve this problem. Here are a few steps that will guide you through this specific scenario: Redirect requests for http://example.com to http://www.example.com and apply it to your Virtual Server (1.2.3.4:80). You can redirect using HTTP Class profiles (v11.3 and prior) or using a policy with Centralized Policy Matching (v11.4 and newer) or you can always write an iRule to redirect! Make www.example.com a CNAME record to example.lb.example.com; where *.lb.example.com is a sub-delegated zone of example.com that resides on your BIG-IP GTM. Create a global traffic pool “aws_us_east” that contains no members but rather a CNAME to us-east.elb.amazonaws.com. Create another global traffic pool “aws_us_west” that contains no members but rather a CNAME to us-west.elb.amazonaws.com. The following screenshot shows the details of creating the global traffic pools (using v11.5). Notice you have to select the "Advanced" configuration to add the CNAME. Create a global traffic Wide IP example.lb.example.com with two pool members “aws_us_east” and “aws_us_west”. The following screenshot shows the details. Create two global traffic regions: “eastern” and “western”. The screenshot below shows the details of creating the traffic regions. Create global traffic topology records using "Request Source: Region is eastern" and "Destination Pool is aws_us_east". Repeat this for the western region using the aws_us_west pool. The screenshot below shows the details of creating these records. Modify Pool settings under Wide IP www.example.com to use "Topology" as load balancing method. See the screenshot below for details. How it all works... Here's the flow of events that take place as a user types in the web address and ultimately receives the correct IP address. External client types http://example.com into their web browser Internet DNS resolution takes place and maps example.com to your Virtual Server address: IN A 1.2.3.4 An HTTP request is directed to 1.2.3.4:80 Your LTM checks for a profile, the HTTP profile is enabled, the redirect request is applied, and redirect user request with 301 response code is executed External client receives 301 response code and their browser makes a new request to http://www.example.com Internet DNS resolution takes place and maps www.example.com to IN CNAME example.lb.example.com Internet DNS resolution continues mapping example.lb.example.com to your GTM configured Wide IP The Wide IP load balances the request to one of the pools based on the configured logic: Round Robin, Global Availability, Topology or Ratio (we chose "Topology" for our solution) The GTM-configured pool contains a CNAME to either us_east or us_west AWS data centers Internet DNS resolution takes place mapping the request to the ELB hostname (i.e. us-west.elb.amazonaws.com) and gives two A records External client http request is mapped to one of the returned IP addresses And, there you have it. With this solution, you can integrate AWS using your existing LTM and GTM technology! I hope this helps, and I hope you can implement this and other solutions using all the flexibility and power of your F5 technology.Use F5 Distributed Cloud to control Primary and Secondary DNS
Overview "Nature is a mutable cloud, which is always and never the same." - Ralph Waldo Emerson We might not wax that philosophically around here, but our heads are in the cloud nonetheless! Join the F5 Distributed Cloud user group today and learn more with your peers and other F5 experts. Scenario 1 – F5 Distributed Cloud DNS: Primary Nameserver Scenario 1.2 - Migrate an Existing Primary DNS Zone to F5 Distributed Cloud DNS Scenario 2 - F5 Distributed Cloud DNS: Primary with Delegated Subdomains Scenario 3 – F5 Distributed Cloud DNS: Secondary Nameserver Conclusion Additional Resources Overview Domain Name Service (DNS); it's how humans and machines discover where to connect. DNS on the Internet is the universal directory of addresses to names. If you need to get support for the product Acme, you go to support.acme.com. Looking for the latest headlines in News, try www.aonn.com or www.npr.org. DNS is the underlying feature that nearly every service on the Internet depends on. Having a robust and reliable DNS provider is critical to keeping your organization online and working, and especially so during a DDoS attack. "Nature is a mutable cloud, which is always and never the same." - Ralph Waldo Emerson We might not wax that philosophically around here, but our heads are in the cloud nonetheless! Join the F5 Distributed Cloud user group today and learn more with your peers and other F5 experts. F5 Distributed Cloud DNS (F5 XC DNS) can function as both Primary or Secondary nameservers, and it natively includes DDoS protection. Using F5 XC DNS, it’s possible to provision and configure primary or secondary DNS securely in minutes. Additionally, the service uses a global anycast network and is built to scale automatically to respond to large query volumes. Dynamic security is included and adds automatic failover, DDoS protection, TSIG authentication support, and when used as a secondary DNS—DNSSEC support. F5 Distributed Cloud allows you to manage all of your sites as a single “logical cloud” providing: - A portable platform that spans multiple sites/clouds - A private backbone connects all sites - Connectivity to sites through its nodes (F5 Distributed Cloud Mesh and F5 Distributed Cloud App Stack) - Node flexibility, allowing it to be virtual machines, live on hardware within data centers, sites, or in cloud instances (e.g. EC2) - Nodes provide vK8s (virtual K8s), network and security services - Services managed through F5 Distributed Cloud’s SaaS base console Scenario 1 – F5 Distributed Cloud DNS: Primary Nameserver Consider the following; you're looking to improve the response time of your app with a geo-distributed solution, including DNS and app distribution. With F5 XC DNS configured as the primary nameserver, you’ll automatically get DNS DDoS protection, and will see an improvement in the response the time to resolve DNS just by using Anycast with F5’s global network’s regional point of presence. To configure F5 XC DNS to be the Primary nameserver for your domain, access the F5 XC Console, go to DNS Management, and then Add Zone. Alternately, if you're migrating from another DNS server or DNS service to F5 XC DNS, you can import this zone directly from your DNS server. Scenario 1.2 below illustrates how to import and migrate your existing DNS zones to F5 XC DNS. Here, you’ll write in the domain name (your DNS zone), and then View Configuration for the Primary DNS. On the next screen, you may change any of the default SOA parameters for the zone, and any type of resource record (RR) or record sets which the DNS server will use to respond to queries. For example, you may want to return more than one A record (IP address) for the frontend to your app when it has multiple points of presence. To do this, enter as many IP addresses of record type A as needed to send traffic to all the points of ingress to your app. Additional Resource Record Sets allows the DNS server to return more than a single type of RR. For example, the following configurations, returns two A (IPv4 address) records and one TXT record to the query of type ANY for “al.demo.internal”. Optionally, if your root DNS zone has been configured for DNSSEC, then enabling it for the zone is just a matter of toggling the default setting in the F5 XC Console. Scenario 1.2 - Migrate an Existing Primary DNS Zone to F5 Distributed Cloud DNS F5 XC DNS can use AXFR DNS zone transfers to import an existing DNS zone. Navigate to DNS Management > DNS Zone Management, then click Import DNS Zone. Enter the zone name and the externally accessible IP of the primary DNS server. ➡️ Note: You'll need to configure your DNS server and any firewall policies to allow zone transfers from F5. A current list of public IP's that F5 uses can be found in the following F5 tech doc. Optionally, configure a transaction signature (TSIG) to secure the DNS zone transfer. When you save and exit, F5 XC DNS executes a secondary nameserver zone AXFR and then transitions itself to be the zone's primary DNS server. To finish the process, you'll need to change the NS records for the zone at your domain name registrar. In the registrar, change the name servers to the following F5 XC DNS servers: ns1.f5clouddns.com ns2.f5clouddns.com Scenario 2 - F5 Distributed Cloud DNS: Primary with Delegated Subdomains An enhanced capability when using Distributed Cloud (F5 XC) as the primary DNS server for your domains or subdomains, is to have services in F5 XC dynamically create their own DNS records, and this can be done either directly in the primary domain or the subdomains. Note thatbeforeJuly 2023, the delegated DNS feature in F5 XC required the exclusive use of subdomains to dynamically manage DNS records. As of July 2023, organizations are allowed to have both F5 XC managed and user-managed DNS resource records in the same domain or subdomain. When "Allow HTTP Load Balancer Managed Records" is checked, DNS records automatically added by F5 XC appear in a new RR set group called x-ves-io-managed which is read-only. In the following example, I've created an HTTP Load Balanacer with the domain "www.example.f5-cloud-demo.com" and F5 XC automatically created the A resource record (RR) in the group x-ves-io-managed. Scenario 3 – F5 Distributed Cloud DNS: Secondary Nameserver In this scenario, say you already have a primary DNS server in your on-prem datacenter, but due to security reasons, you don’t want it to be directly accessible to queries from the Internet. F5 XC DNS can be configured as a secondary DNS server and will both zone transfer (AXFR, IXFR) and receive (NOTIFY) updates from your primary DNS server as needed. To configure F5 XC DNS to be a secondary DNS server, go to Add Zone, then choose Secondary DNS Configuration. Next, View Configuration for it, and add your primary DNS server IP’s. To enhance the security of zone transfers and updates, F5 XC DNS supports TSIG encrypted transfers from the primary DNS server. To support TSIG, ensure your primary DNS server supports encryption, and enable it by entering the pre-shared key (PSK) name and its value. The PSK itself can be blindfold-encrypted in the F5 XC Console to prevent other Console admins from being able to view it. If encryption is desired, simply plug in the remaining details for your TSIG PSK and Apply. Once you’ve saved your new secondary DNS configuration, the F5 XC DNS will immediately transfer your zone details and begin resolving queries on the F5 XC Global Network with its pool of Anycast-reachable DNS servers. Conclusion You’ve just seen how to configure F5 XC DNS both as a primary DNS as well as a secondary DNS service. Ensure the reachability of your company with a robust, secure, and optimized DNS service by F5. A service that delivers the lowest resolution latency with its global Anycast network of nameservers, and one that automatically includes DDoS protection, DNSSEC, TSIG support for secondary DNS. Watch the following demo video to see how to configure F5 XC DNS for scenarios #1 and #3 above. Additional Resources For more information about using F5 Distributed Cloud DNS: https://www.f5.com/cloud/products/dns For technical documentation: https://docs.cloud.f5.com/docs/how-to/app-networking/manage-dns-zones DNS Management FAQ: https://f5cloud.zendesk.com/hc/en-us/sections/7057223802519-DNS-Management DNS Demo Guide and step-by-step walkthrough: https://github.com/f5devcentral/f5xc-dnsSSL Orchestrator Advanced Use Cases: DNS-over-HTTPS Detection
Introduction F5 BIG-IP is synonymous with "flexibility". You likely have few other devices in your architecture that provide the breadth of capabilities that come native with the BIG-IP platform. And for each and every BIG-IP product module, the opportunities to expand functionality are almost limitless. In this article series we examine the flexibility options of the F5 SSL Orchestrator in a set of "advanced" use cases. If you haven't noticed, the world has been steadily moving toward encrypted communications. Everything from web, email, voice, video, chat, and IoT is now wrapped in TLS, and that's a good thing. The problem is, malware - that thing that creates havoc in your organization, that exfiltrates personnel records to the Dark Web - isn't stopped by encryption. TLS 1.3 and multi-factor authentication don't eradicate malware. The only reasonable way to defend against it is to catch it in the act, and an entire industry of security products are designed for just this task. But ironically, encryption makes this hard. You can't protect against what you can't see. F5 SSL Orchestrator simplifies traffic decryption and malware inspection, and dynamically orchestrates traffic to your security stack. But it does much more than that. SSL Orchestrator is built on top of F5's BIG-IP platform, and as stated earlier, is abound with flexibility. Let us now explore one perspective of that flexibility and how SSL Orchestrator can be used to handle DNS-over-HTTPS and DNS-over-TLS. SSL Orchestrator Use Case: DNS-over-HTTPS and DNS-over-TLS Handling Despite the rapid evolution of Internet standards, and increasing amount of encryption, there's one aspect of our daily online world that hasn't really changed that much in its nearly 4 decades of breath. That of course is DNS. We don't tend to think about it often, which is probably why it hasn't evolved as much as other things, but it truly is the heart and backbone of everything we do online. That is, unless you want to memorize "2607:f8b0:400a:0804:0000:0000:0000:2004" as the way to get to Google, you had better have a working DNS. But DNS is inherently insecure. It's been shown to be vulnerable to all manner of attacks, and for the purposes of this discussion specifically, also exposes where you're going. That is, while the HTTP payload may be encrypted, there's still that (visible) DNS request that goes out first. That's not to say that there haven't been any improvements though. DNSSEC was developed to help secure DNS and prevent spoofing, but in the many years since its introduction it still isn't as widespread as we'd have hoped. DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT) are more recent developments that focus mainly on the privacy aspect of DNS communications (or lack thereof). With DoH and DoT, clients and servers forego the typical DNS protocol request over UDP or TCP port 53 and embed the request inside an encrypted HTTPS or pure TLS connection. But...while that sounds pretty cool, there can be additional consequences to encrypting DNS, both good and bad: [Good] DoH and DoT can protect against ISPs that use your DNS information for targeted advertising. [Bad] DoH and DoT can effectively blind local DNS security and filtering controls. DNS monitoring is often an effective defense against spam and malware infections. [Bad] DoH and DoT providers are public services (ex. Cloudflare and Google), so while DNS is being protected from eavesdropping along the way, the providers have access to your DNS requests. The US National Security Agency (NSA) actually published a warning on this: https://media.defense.gov/2021/Jan/14/2002564889/-1/-1/0/CSI_ADOPTING_ENCRYPTED_DNS_U_OO_102904_21.P... The basic idea in this document is that while your DNS privacy is a good thing, encrypted DoH and DoT can also mask malware command-and-control. If you strictly follow the NSA guidance, you could either block DoH and/or DoT altogether, or force users to only use a local enterprise DoH resolver. If you don't have access to a local DoH resolver though, what else could you do? Well, I'm so glad you asked. DoH and DoT are of course encrypted, so unless you disable either protocol at the client, set up a local DoH resolver and force clients to it, or attempt to do encrypted analysis to find DoH traffic, you need to decrypt and inspect. And as it turns out, F5 has an elegant solution to do just that. In this blog post I am going to present a few solutions on how to handle these protocols through SSL Orchestrator decrypted analysis. The protocol implementations of DoH and DoT are a little different, so I will address them separately. Handling DNS-over-HTTPS For all intents and purposes, DoH looks and feels like normal HTTPS traffic. There are some semi-unique patterns you might be able to follow to infer that something is DoH rather than regular HTTPS (i.e., packet size, frequency), but it’s never going to be completely accurate. You could also potentially filter on the IP addresses of the known public DoH services, but that list is rather large and growing: https://github.com/curl/curl/wiki/DNS-over-HTTPS, and clients can ultimately choose the service they point to. If you’re serious about actually detecting DoH traffic with reasonable accuracy, then there’s no better way than through decrypted analysis. Now, once you've decrypted HTTPS and detected that it's DoH, there are a number of things you can do: Simply detect and block: there are actually three forms of DoH requests – Wireformat GET and POST, and JSON (detailed here:https://developers.cloudflare.com/1.1.1.1/dns-over-https), though the vast majority of clients use the Wireformat GET and POST methods. In any case, this option follows the NSA's first recommendation to simply detect and block DoH requests. A browser that receives a reject on a DoH request will natively revert to normal TCP/UDP DNS to allow your local DNS security tools to do what they do best. Detect and log: you may simply want to log that DoH is passing through your network, who's asking, and what they're asking for as an extension to your current DNS monitoring protocols. Detect and blackhole: also, as a possible extension to your current DNS protection protocols, you may need to block some DoH requests from happening. As mentioned earlier, if you simply block DoH, the client will revert to DNS. One of the best ways to block a DoH/DNS request is to provide a good but fake response instead, a technique called "blackholing". In this approach, you can flag specific hostnames (matching on URL categories) to provide a blackhole response. Let’s now look at each of the above options in turn. The first option is actually pretty straightforward. You just need to add the below iRule to an SSL Orchestrator outbound layer 3 topology*, on the “-in-t” TCP tunnel virtual server. On any decrypted HTTPS traffic, the iRule examines the HTTP request methods for the three signature DoH methods and sends a reject on a match. Easy peasy. when HTTP_REQUEST priority 750 { if { ( [HTTP::method] equals "GET" and [HTTP::header exists "accept"] and [HTTP::header "accept"] equals "application/dns-json" ) or \ ( [HTTP::method] equals "GET" and [HTTP::header exists "accept"] and [HTTP::header "accept"] equals "application/dns-message" ) or \ ( [HTTP::method] equals "POST" and [HTTP::header exists "content-type"] and [HTTP::header "content-type"] equals "application/dns-message" ) } { reject } } The second two options above are available in a single iRule here:https://github.com/f5devcentral/sslo-script-tools/tree/main/sslo-dns-over-https-detection. As with the previous, import the iRule to the BIG-IP, then associate that iRule with an SSL Orchestrator outbound layer 3 topology*. You'll need to make a few adjustments in the iRule to suit for your environment, and those are all presented as simple static variable assignments: static::LOCAL_LOG: enable or disable local Syslog logging (0 = off, 1 = on). static::HSL: enable or disable remote high-speed logging (0 = off, 1 = on). static::URLDB_LICENSED: set to 1 (on) if you've licensed and provisioned the subscription based URLDB, or 0 (off) if you have not and want to just use custom URL categories for DoH blackholing. static::BLACKHOLE_URLCAT: leave empty to disable or add a list of URL categories to search for URL-based DoH blackholing (ex. /Common/block-doh-urls) static::BLACKHOLE_RTYPE_A: enable or disable blackholing matched A record requests (0 = off, 1 = on). static::BLACKHOLE_RTYPE_AAAA: enable or disable blackholing matched AAAA record requests (0 = off, 1 = on). static::BLACKHOLE_RTYPE_TXT: enable or disable blackholing matched TXT record requests (0 = off, 1 = on). You can quickly test the solution by enabling DoH support in a browser. Please refer to the following instructions for each: https://developers.cloudflare.com/1.1.1.1/encryption/dns-over-https/encrypted-dns-browsers/. With the static::LOCAL_LOG value set to 1 (enabled), you can tail the BIG-IP LTM log (tail -f /var/log/ltm) and watch the DoH traffic flow across the SSL Orchestrator topology. 10.1.10.50:59804-104.18.42.171:443 :: DoH (WireFormat POST) Request: A:www.google.com 10.1.10.50:59804-104.18.42.171:443 :: DoH (WireFormat POST) Request: A:encrypted-tbn0.gstatic.com 10.1.10.50:59804-104.18.42.171:443 :: DoH (WireFormat POST) Request: A:fonts.gstatic.com 10.1.10.50:59812-104.18.42.171:443 :: DoH (WireFormat POST) Request: A:www.gstatic.com 10.1.10.50:59812-104.18.42.171:443 :: DoH (WireFormat POST) Request: A:id.google.com 10.1.10.50:59812-104.18.42.171:443 :: DoH (WireFormat POST) Request: A:play.google.com Handling DNS-over-TLS DNS-over-TLS (DoT) is essentially plain DNS wrapped in TLS. If you decrypt it, you’ll see a regular DNS request. DoT by standard, travels on TCP port 853, so that’s one very simple way to block DoT traffic, by just blocking that port. But if you wanted to actually log the DoT requests flowing through the BIG-IP, or blackhole some URLs, you again have to decrypt. The aforementioned DoH logging iRule can also handle DoT requests, inspecting any decrypted TCP:853 traffic: https://github.com/f5devcentral/sslo-script-tools/tree/main/sslo-dns-over-https-detection. The same static variable flags apply. Considering DoH/DoT proxy options I would be remiss in not mentioning options to actually proxy DoH and DoT traffic, and BIG-IP 16.1 introduced two new capabilities: DoH proxy – where the BIG-IP proxies client DoH requests to external DoH resources. You get to explicitly decrypt here, so you get another opportunity to log and control the DNS. DoH server – where the BIG-IP proxies client DoH requests to external TPC or UDP DNS resources. As with the proxy option, you get to decrypt the client’s DoH request. You can find out more about both of these capabilities here: https://support.f5.com/csp/article/K05451012 To proxy DoT to DoT, you simply need a TCP:853 virtual server with client and server SSL profiles. There are also various programmatic options to convert DoT to DNS, DNS to DoT, and DNS to DoH. But an important consideration here is that to proxy you would need clients to direct their DNS/DoT/DoH requests to your local resource. In some cases, you can do that through DHCP, or through enterprise browser policy management, but other than simply blocking outbound access to these protocols, it can be non-trivial to prevent clients from trying to use any of the external providers. That is specifically where decrypted analysis can be beneficial. Summary With an iRule and just a few configuration changes we have been able to implement a capability on top of SSL Orchestrator to log and actively control encrypted DNS. Decrypted inspection of DoH and DoT is just one of the many interesting benefits of and SSL Orchestrator solution. A demo of the above is available here:SSL Orchestrator Advanced Use Case: DNS over HTTPS * The iRules will also technically work for outbound layer 2 topologies.
BIG-IP Advanced Firewall Manager (AFM) DNS NXDOMAIN Query Attack Type Walkthrough - part two
This is part two of theBIG-IPAdvanced Firewall Manager (AFM) DNS NXDOMAIN Query Attack Type Walkthrough article series. Part one is athttps://community.f5.com/t5/technical-articles/big-ip-advanced-firewall-manager-afm-dns-nxdomain-query-attack/ta-p/317656 Reviewing Part One In part one, we looked at: What and How of DNS NXDOMAIN response and flood definitions How threat actors generate random dns queries with the use of Domain Generation Algorithms and concepts such as DNS Blackhole, Fast Flux,DNS water torture attack What information can be used in observing NXDOMAIN response spike ReviewingBIG-IP DNS profile statistics collected using a periodic data collection script - provides visibility/statistics on the type of requests and responses the DNS listener processed which are useful when reviewing a recent DNS traffic spike event and understanding the characteristic of the traffic. Reviewing sample packet capture during a NXDOMAIN response spike and reducing and zooming in to the data of interest using commands such as capinfos, tshark, sort, uniq, wc Configuring Detection and Mitigation Thresholds In this article, we will continue using the information gathered from the NXDOMAIN response flood packet capture and configure BIG-IP Advanced Firewall Manager (AFM) DNS NXDOMAIN Query Attack Type Detection and Mitigation Thresholds. Configuring BIG-IP AFM DNS NXDOMAIN attack type to mitigate NXDOMAIN response spike Now that we have information from the sample packet capture and extracted DNS names we can start working on using these information to configure BIG-IP AFM NXDOMAIN attack type Detection and Mitigation thresholds. The sample packet capture we reviewed ran approx. 13 mins and recorded 60063 packets. If we divide the number of packets to number of seconds the pcap ran - 60063 p/780s, the number of packets per second is 334 packets per second. Since the packet capture contains only DNS traffic, we can expect it to have both dns requests and responses, which further reduces to 167 packets per second for either dns request or response. Since NXDOMAIN is a response and the packet capture was taken in a simulated attack to produce NXDOMAIN response, we can use this 167 packets per second as a baseline of what attack traffic looks like. We should aim for a lower number of packets per second to detect the attack and provide an allowance before we start mitigating the NXDOMAIN response flood. For the purpose of the demonstration, I have configured a lower detection and mitigation threshold to mitigate the NXDOMAIN response flood. This configuration is on a DNS enabled AFM DoS protection profile that will be applied to a Virtual Server. Dns-dos-protect is the name of the profile in this lab test. Configuring BIG-IP AFM DNS NXDOMAIN query attack type in a DNS enabled AFM DoS protection profile Detection Threshold: 20 EPS Mitigation Threshold: 30 EPS I'll be using 2 test clients to send a flood dns queries to a DNS listener for the hostnames generated thru DGA. As expected, the response for these queries will be NXDOMAIN and the AFM DNS NXDOMAIN Query Attack type will detect the attack as soon as 20 NXDOMAIN responses are observed and will start to drop excess of 30 NXDOMAIN responses. Here is a sample script to read a file , line by line, that contains DNS names to query. 10.93.56.197 is the DNS listener where the DNS enabled AFM DoS protection profile "dns-dos-protect" is applied to. root@ubuntu-server1:~# cat nxdig.sh #!/bin/bash while read -r line; do dig @10.93.56.197 "$line" done wait Here is the DNS listener DoS protection profile configuration. It also shows dos-dns-logging-profile is used as a Log profile. Here is the dos-dns-logging-profile Log profile profile configuration which only have DNS DoS protection logs enabled and logging to the local-db-publisher (logdb, a mysql db in the BIG-IP) Using the tmsh show security dos profile <DoS Protection profile name>, we can view the statistics observed by the DoS protection profile per Attack Type. In this test, only NXDOMAIN Query is enabled. Using the same periodic data capture script when we observed the ltm dns profile statistics, we can capture statistics for the DoS protection profile for review and understanding the phases of the attack being observed. while true; do date >> /var/tmp/afm_nx_stats; tmsh show security dos profile dns-dos-protect >> /var/tmp/afm_nx_stats; echo "###################" >> /var/tmp/afm_nx_stats; sleep 2; done Run the dns query flood using the script while true; do ./nxdig.sh < nx.txt 2>&1; done Here is a screenshot when the detection threshold of the NXDOMAIN Query attack type was exceeded, see the Attack Status. Here is when the Attack is being Mitigated Here is when there is no more Attack being detected A look at the periodic data capture for the DoS protection profile shows interesting statistics. Attack Detected - value of 1 means attack is detected which also means the detection threshold was exceeded, value of 0 means no attack currently detected. Stats - number of packets observed by the Attack type - this is a cumulative value since the BIG-IP booted up Stats Rate - current number of packets observed by the Attack type - provides an idea of how much of this type of packets currently observed, you can think of this as the current - Events Per Second - EPS of the attack type Stats 1m - average number of packets per second observed by the Attack type in the last 1m - provides an idea of average number packets of this type every second for the last 1m The Stats Rate and Stats 1m can provide an idea of how much packets can be seen in the current second and the average per second in the last minute. In non-attack scenario, observing these values shows what normal number of packets may look like. During an attack scenario - detection threshold exceeded - it can provide an idea how much the attack type was seeing. These information can then be used as a basis for setting the mitigation threshold. For example, it was observed that the Stats 1m value was 21 and during an attack scenario, the Stats Rate value was 245, this is about 12x of the average and the volume appears to be an attack. Depending on the risk appetite of the business, an allowance of 2x of the average number of packets for the attack type is where they want to drop exceeding packets , thus, 42 EPS can be configured for the mitigation threshold. Do note, setting low detection and mitigation thresholds can cause false positives, triggering detection and mitigation too early. Therefore it is important to understand the traffic characteristic for an attack type. In the gathered data, we can see here Attack Detected is 1, which means the detection threshold was exceeded. Stats Rate is at 245 which do exceed the 20 EPS detection threshold. Note that there were no Drops stats yet. Tue Jun 27 07:47:52 PDT 2023 | Attack Detected 1 | .. | Aggregate Attack Detected 1 | Attack Count 1 .... | Stats 1461 | Stats Rate 245 | Stats 1m 21 | Stats 1h 0 | Drops 0 4 seconds later, we do see the Drops count is 2, which tells us the Mitigation threshold - configured as 30 EPS - was exceeded. Tue Jun 27 07:47:56 PDT 2023 Attack Detected 1 ... | Aggregate Attack Detected 1 | Attack Count 1 | Stats 1h Samples 0 | Stats 1516 | Stats Rate 0 | Stats 1m 20 | Stats 1h 0 | Drops 2 3 seconds later, we do see the Drops count is 4, which tells us the mitigation is ongoing and dropping excess packets. Tue Jun 27 07:47:59 PDT 2023 | Attack Detected 1 ... | Aggregate Attack Detected 1 | Attack Count 1 | Stats 1h Samples 0 | Stats 1578 | Stats Rate 0 | Stats 1m 20 | Stats 1h 0 | Drops 4 From these sample stats, particularly the 'Stats' value, in the 7 seconds - 07:47:52 to 07:47:59 - the difference is 117 (1578 - 1461), which tells us that the packet of this type volume is low - averages at 17 packets per second for the last 7 seconds.If the difference on these 'Stats' values are much bigger, then we potentially have traffic spike. Drops stats increasing means that an attack is still being mitigated and the volume of the packets is not yet lower that the defined mitigation threshold.Drops - number of packets observed by the Attack type - this is a cumulative value since the BIG-IP booted upDrops Rate - current number of packets dropped by the Attack type Reviewing DoS stats information in the Reporting DoS DashboardWe have seen the DoS protection profile stats output, now we switch to the GUI and review the same DoS stats information.In the Reporting DoS Dashboard, there are records of the recorded Attacks. The timeframe can be adjusted to find incident of interest. In the testing done, I filtered DNS only and Attack IDs are displayed along with very useful information and statistics. In this screenshot, Attack ID 2958374472 was selected and relevant statistics are displayed. It was of DNS NXDOMAIN query Attack type/vector it shows how much packets were observed in this attack, which is 235 packets, and dropped packets at 25. Avg PPS - average packets per second for the duration of the attack, similar to the Stats Rate 1m, can also be used as a basis for the Detection and mitigation threshold of the attack type. Domain names observed are also recorded along with the same statistics on the attack. Configuring BIG-IP AFM DNS NXDOMAIN query attack type in AFM Device DoS Protection AFM Device Protection also have the DNS NXDOMAIN query attack type. This is a device wide protection and protects the self Ips and Virtual Servers of the BIG-IP. Detection and Mitigation thresholds can be configured the same way, observed the traffic type using the same type of statistics - but this time its Device DoS protection specific. Here is a sample of tmshshow security dos device-config dos-device-configoutput and piping it to grep to filter specifically lines for DNS NXDOMAIN Query. tmsh show security dos device-config dos-device-config | grep -i nxdomain -A 40 Security::DoS Config: DNS NXDOMAIN Query ------------------------------------------------------ Statistics Type Count Detection Method Static Vector - Inline Status Ready Attack Detected 1 ... Aggregate Attack Detected 1 Attack Count 20 Stats 1h Samples 0 Stats 46580 Stats Rate 187 Stats 1m 73 Stats 1h 8 ... Drops 150 ...snip.. AutoDetection 137 Mitigation Low 4294967295 Similarly, in the GUI, we can observe the states of an attack detected and mitigated by the NXDOMAIN query attack type configured in the AFM Device DoS protection. Here is the Detection and Mitigation threshold configuration Attack Detected Attack being Mitigated We can review the Reporting DoS Dashboard of the Attack events for the Device DoS and review the statistics Configuring Valid FQDNs in the DNS NXDOMAIN Query Attack type TheDNS NXDOMAIN Query Attack type has a configuration calledValid FQDNs and is described as: Allows you to create a whitelist of valid fully qualified domain names. In theAdd new FQDNfield, type a domain name and clickCheckto see if it is already on the list, clickAddto add it to the list, or clickDeleteto remove it from the whitelist. Take the name qehspqnmrn[.]fop789[.]loc as an example. Let's assume that this is a valid DNS hostname/FQDN and we do not wantDNS NXDOMAIN Query Attack type to drop packets for its response even though it would result in a NXDOMAIN response. We can add it in theValid FQDNslist from the GUI or tmsh. Here is the tmsh example, sinceqehspqnmrn[.]fop789[.]loc is already in the Valid FQDNs list, let's add another FQDN site1[.]fop789[.]loc. tmsh modify security dos device-config dos-device-config dos-device-vector { dns-nxdomain-query { valid-domains add { site1[.]fop789[.]loc } } } Here is the tmsh output when listing the DNS NXDOMAIN Query Attack Type including the Valid FQDNs. tmsh list security dos device-config dos-device-config dos-device-vector { dns-nxdomain-query } security dos device-config dos-device-config { dos-device-vector { dns-nxdomain-query { ... valid-domains { qehspqnmrn[.]fop789[.]loc site1[.]fop789[.]loc } } } } To verify that the packets for the FQDN in the Valid FQDNs list are not being dropped, we can look at theReporting DoS Dashboard. We can see ongoing attacks are reported and also lists the Domain Names in the attack. Taking a closer look at the statistics,qehspqnmrn[.]fop789[.]loc in the Domain Name list has NO packet Drops and NO Attack detected. The rest of the DNS names in the list have Drops and Attacks and are being mitigated by the AFM DNS NXDOMAIN query attack type.BIG-IP Advanced Firewall Manager (AFM) DNS NXDOMAIN Query Attack Type Walkthrough
Introduction In this article, we will look at configuring BIG-IP Advanced Firewall Manager's (AFM) DNS NXDOMAIN attack type in the Device Protection and DNS enabled protection profile to mitigate DNS NXDOMAIN response floods. We will review data from a packet capture and BIG-IP DNS' DNS profile statistics to set detection and mitigation thresholds. This is part one of two of this article series. Part two is athttps://community.f5.com/t5/technical-articles/big-ip-advanced-firewall-manager-afm-dns-nxdomain-query-attack/ta-p/317681 What is a NXDOMAIN dns response The DNS protocol [RFC1035] defines response code 3 as "Name Error", or "NXDOMAIN" [RFC2308], which means that the queried domain name does not exist in the DNS. Since domain names are represented as a tree of labels ([RFC1034], Section3.1), nonexistence of a node implies nonexistence of the entire subtree rooted at this node. From https://datatracker.ietf.org/doc/html/rfc8020#page-5 How is NXDOMAIN dns response generated RCODE Response code - this 4 bit field is set as part of responses. The values have the following interpretation: 3 Name Error - Meaningful only for responses from an authoritative name server, this code signifies that the domain name referenced in the query does not exist. https://datatracker.ietf.org/doc/html/rfc1035#section-4.1.1 What is a NXDOMAIN (response) flood? From the F5 Glossary in https://www.f5.com/glossary/dns-flood-nxdomain-flood The roadmap to every single computer on the Internet is held in DNS servers. The DNS NXDOMAIN flood attack attempts to make servers disappear from the Internet by making it impossible for clients to access the roadmap. In this attack, the attacker floods the DNS server with requests for invalid or nonexistent records. The DNS server spends its time searching for something that doesn't exist instead of serving legitimate requests. The result is that the cache on the DNS server gets filled with bad requests, and clients can't find the servers they are looking for. How do threat actors generate random dns queries There are many tools to generate a flood of DNS queries . The DNS records in the flood of DNS queries for the most part, should be unique if an attacker wants to poison a DNS servers cache. Otherwise, a DNS administrator can simply blackhole if the same DNS record is queried in DNS flood query. DNS Blackhole https://en.wikipedia.org/wiki/DNS_sinkhole Here is a devcentral article on DNS Blackhole implemented in an iRule https://community.f5.com/t5/codeshare/dns-blackhole/ta-p/283786 Domain Generation Algorithms To generate many random host records for one or many domains, Domain Generation Algorithms are used. https://en.wikipedia.org/wiki/Domain_generation_algorithm Many examples of DGA as collected in https://github.com/baderj/domain_generation_algorithms shows how these random records may look like. In the lab setup we will be using, we have the fop789[.]loc domain, and I borrowed from the github page random host list generated thru the "mydoom (aka Novarg, Mimail.R, Shimgapi)" DGA and appended the host part of the host list to the test lab domain . qehspqnmrn[.]fop789[.]loc mmahaesqar[.]fop789[.]loc pwprhhnqqn[.]fop789[.]loc .... Here is a sample query using the one of the hostnames. As expected, a NXDOMAIN response is received because this record does not exist in the sample lab domain fop789.loc. root@ubuntu-server1:~# dig @10.93.56.197 qehspqnmrn.fop789.loc ; <<>> DiG 9.16.1-Ubuntu <<>> @10.93.56.197 qehspqnmrn.fop789.loc ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 1279 ;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;qehspqnmrn.fop789.loc. IN A ;; AUTHORITY SECTION: fop789.loc. 300 IN SOA ns1.fop789.loc. hostmaster.ns1.fop789.loc. 2023041101 10800 3600 604800 86400 ;; Query time: 4 msec ;; SERVER: 10.93.56.197#53(10.93.56.197) ;; WHEN: Sun Jun 25 11:10:41 UTC 2023 ;; MSG SIZE rcvd: 101 root@ubuntu-server1:~# The most prevalent reason why threat actors would use DGA is in malware and phishing campaigns to avoid detection and resilient to counter measures. The evasion technique used is Fast flux. Fast flux Fast flux is a domain name system (DNS) based evasion technique used by cyber criminals to hide phishing and malware delivery websites behind an ever-changing network of compromised hosts acting as reverse proxies to the backend botnet master—a bulletproof autonomous system. It can also refer to the combination of peer-to-peer networking, distributed command and control, web-based load balancing and proxy redirection used to make malware networks more resistant to discovery and counter-measures. The fundamental idea behind fast-flux is to have numerous IP addresses associated with a single fully qualified domain name, where the IP addresses are swapped in and out with extremely high frequency, through changing DNS resource records, thus the authoritative name servers of the said fast-fluxing domain name is—in most cases—hosted by the criminal actor Depending on the configuration and complexity of the infrastructure, fast-fluxing is generally classified into single, double, and domain fast-flux networks. Fast-fluxing remains an intricate problem in network security and current countermeasures remain ineffective. https://en.wikipedia.org/wiki/Fast_flux What information can be used in observing NXDOMAIN response spike There are several sources of information that can be used when NXDOMAIN response spike. BIG-IP DNS profile statistics A BIG-IP DNS listener (Virtual Server) will have a DNS profile applied to it. This profile provides access to DNS traffic statistics. In particular, "Question Type" and "Return Code" sections has statistics on DNS record types queried and return code count. Here is a sample output from a script that periodically captured DNS profile statistics - stats were taken 20 seconds apart. These were taken during a lab test where NXDOMAIN response flood is being simulated. Here is the sample script: while true; do date >> /var/tmp/dns_stat.txt; tmsh show ltm profile dns dns-prof-1 >> /var/tmp/dns_stat.txt; echo "###################" >> /var/tmp/dns_stat.txt; sleep 20; done Notice that the "Question Type" has only "A" records queried and in the "Return Code (RCODE)" , only " No Name (NXDOMAIN)" were the responses. Date: Sun Jun 25 11:15:59 PDT 2023 --------------------------------------------------- Ltm::DNS Profile: dns-prof-1 --------------------------------------------------- Virtual Server Name N/A Query Message Recursion Desired (RD) 18847 100.0 DNSSEC Checking Disabled (CD) 0 0.0 EDNS0 18847 100.0 Client Subnet 0 0.0 Client Subnet Inserted 0 0.0 Operation Code (OpCode) Query 18847 100.0 Notify 0 0.0 Update 0 0.0 Other 0 0.0 Question Type A 18847 100.0 AAAA 0 0.0 ANY 0 0.0 CNAME 0 0.0 MX 0 0.0 ... Other 0 0.0 Response Message Authoritative Answer (AA) 18843 99.9 Recursion Available (RA) 0 0.0 Authenticated Data (AD) 0 0.0 Truncated (TC) 0 0.0 Return Code (RCODE) No Error 1 0.0 No Name (NXDOMAIN) 18842 99.9 Server Failed 0 0.0 Refused 1 0.0 Bad EDNS Version 0 0.0 Name Error (NXDOMAIN) Override 0 0.0 EDNS0 client subnet 0 0.0 Date: Sun Jun 25 11:16:19 PDT 2023 Query Message Recursion Desired (RD) 18993 100.0 DNSSEC Checking Disabled (CD) 0 0.0 EDNS0 18993 100.0 Client Subnet 0 0.0 Client Subnet Inserted 0 0.0 Operation Code (OpCode) Query 18993 100.0 Notify 0 0.0 Update 0 0.0 Other 0 0.0 Question Type A 18993 100.0 AAAA 0 0.0 ANY 0 0.0 CNAME 0 0.0 ... Other 0 0.0 Response Message Authoritative Answer (AA) 18989 99.9 Recursion Available (RA) 0 0.0 Authenticated Data (AD) 0 0.0 Truncated (TC) 0 0.0 Return Code (RCODE) No Error 1 0.0 No Name (NXDOMAIN) 18988 99.9 Server Failed 0 0.0 Refused 1 0.0 Bad EDNS Version 0 0.0 Name Error (NXDOMAIN) Override 0 0.0 EDNS0 client subnet 0 0.0 Sample packet capture during a NXDOMAIN response spike Earlier, we reviewed what is Domain Generation Algorithm (DGA) and that its used to generate random DNS names which are used by threat actors in fast flux technique to evade detection and mitigations for their phishing and malware campaigns. The sample packet capture was taken while using the sample random DNS names generated thru a DGA to simulate a NXDOMAIN response flood. Using the capinfos command, we can observe various details about the packet capture. First and last packet time tells us how long this pcap was running, in this case it has been around 13 mins. Also average packet rate per second, 76 packets/sec, can be useful if we are looking to find a baseline on packets per second value. And other packet capture details which may be useful depending on your purpose. Since we are looking at DNS traffic, remember its query and response packets, so 76 packets per second, presumably may contain both type of packets, thus, estimation of 38 packets per second for dns queries. [root@bigip:TimeLimitedModules::Active:Standalone] tmp # capinfos nx-4.pcap File name: nx-4.pcap File type: Wireshark/tcpdump/... - pcap File encapsulation: Ethernet File timestamp precision: microseconds (6) Packet size limit: file hdr: 65535 bytes Number of packets: 60 k File size: 16 MB Data size: 15 MB Capture duration: 786.961838 seconds First packet time: 2023-06-25 11:05:48.352417 Last packet time: 2023-06-25 11:18:55.314255 Data byte rate: 19 kBps Data bit rate: 159 kbps Average packet size: 260.57 bytes Average packet rate: 76 packets/s SHA1: 10d3652ce3b97d68d16f324ee6eaac918b8f34d9 RIPEMD160: 7c766e4e5819fb9f5c90cee133f0e1f61e9b5801 MD5: 51cd10815de35802460ae0b26e156d66 Strict time order: False Number of interfaces in file: 1 Interface #0 info: Encapsulation = Ethernet (1 - ether) Capture length = 65535 Time precision = microseconds (6) Time ticks per second = 1000000 Number of stat entries = 0 Number of packets = 60063 Next up, use tshark to get more information from the packet capture. Specifically, we are interested in the dns related information, such as the DNS records queried and the DNS response. Extract all dns names from the packet capture - queries or response [root@bigip:TimeLimitedModules::Active:Standalone] tmp # tshark -r nx-4.pcap -n -T fields -e dns.qry.name > nx-4-pcap-records.txt Running as user "root" and group "root". This could be dangerous. Reviewing the number of dns names extracted, it matches the output from capinfos - 60063. [root@bigip:TimeLimitedModules::Active:Standalone] tmp # cat nx-4-pcap-records.txt | wc -l 60063 Sort the dns names extracted from the pcap - notice the randomness of these dns names. Could be fast flux technique or an attempt to drown a DNS server of random records that it needs to search and eventually cause service disruption - a classic DNS water torture attack. The sample DNS names taken from the pcap are still not so random and are short. There are longer and more random hostnames that can be generated by DGAs and this can really take a lot of memory and cpu resource from a DNS server. [root@bigip:TimeLimitedModules::Active:Standalone] tmp # cat nx-4-pcap-records.txt | sort | uniq -c | sort -nrk 1 3514 mmahaesqar.fop789.loc 3512 arphansaqh.fop789.loc 3509 hwepmerswa.fop789.loc 3508 qrqnswerqs.fop789.loc 3506 seenwrqrps.fop789.loc 3506 pwprhhnqqn.fop789.loc 3506 hrhspsrenn.fop789.loc 3506 arwrseqssh.fop789.loc 3505 eqqhnpswmh.fop789.loc 3504 fop789.locaehwmnms.fop789.loc 3503 qehspqnmrn.fop789.loc 3503 psrhaaeqqa.fop789.loc 3503 paepnpamea.fop789.loc 3503 ewamspqwha.fop789.loc 3501 aepaaemrmn.fop789.loc 3500 mrspmramrn.fop789.loc 3499 rnqhapapwn.fop789.loc 475 DNS Water Torture Denial-of-Service Attacks Customers reported pseudo-random subdomain or “DNS water torture attacks” hitting their networks with half a million connections per second. Outages were occurring even if a network wasn’t the direct target of the attack. For example, service providers still felt the effects as the DNS water torture traffic passed through their networks and saturated their pipes. To pull off a DNS water torture attack, an attacker leverages a botnet (or thingbot) to make thousands of DNS requests for fake subdomains against an Authoritative Name Server.1 Because the requests are for non-existent subdomains or hosts, the requests consume the memory and processing resources on the main resolver. If there are intermediary DNS resolvers inline, they too get clogged up with these fake requests. For legitimate end users, all this resource consumption means everything runs slow or even stops, resulting in a denial of service. https://www.f5.com/labs/articles/threat-intelligence/the-dns-attacks-we-re-still-seeing#:~:text=To%20pull%20off%20a%20DNS,against%20an%20Authoritative%20Name%20Server.&text=Because%20the%20requests%20are%20for,resources%20on%20the%20main%20resolver. Filtering further, we can extract the DNS response packets only [root@bigip:TimeLimitedModules::Active:Standalone] tmp # tshark -r nx-4.pcap -n -Y "dns.flags.response == 1" -T fields -e dns.qry.name > nx-4-pcap-records-response.txt Running as user "root" and group "root". This could be dangerous. DNS response packets only count shows 29994, approx half of the previous output of 60063 [root@bigip:TimeLimitedModules::Active:Standalone] tmp # cat nx-4-pcap-records-response.txt | wc -l 29994 We can then sort DNS response packets only and find the number of times each DNS name was responded to. We know that the response to these queries are NXDOMAIN because we don’t have these records in the lab DNS server records list. We can also observe which DNS names were responded the most. In the example output, each of the records were responded almost equally. [root@bigip:TimeLimitedModules::Active:Standalone] tmp # cat nx-4-pcap-records-response.txt | sort | uniq -c | sort -nrk 1 1748 qehspqnmrn.fop789.loc 1748 mmahaesqar.fop789.loc 1746 seenwrqrps.fop789.loc .... 1746 aepaaemrmn.fop789.loc 1744 paepnpamea.fop789.loc 1744 arphansaqh.fop789.loc 1742 eqqhnpswmh.fop789.loc 316
Protecting Beyond DNS Flood & DDoS
The recent slate of cyber-attacks involving DNS and NTP systems has again prompted questions about the comprehensiveness of DNS infrastructure’s security protection. Besides mitigating volumetric attacks such as DNS flood & DDoS, many organizations have realized the need for a more comprehensive DNS security protection, which helps in preventing DNS-related security frauds and non-volumetric based attacks such as amplification and cache poisoning attacks. On DNS Amplification & DNS Reflection Attacks You might concur that increasing DNS performance with adequate DNS rate limiting mechanism is probably one of the best approaches to tackle the problem of overwhelming DNS traffic and DNS DoS attacks. However, this does not address the issue of DNS Amplification and DNS reflection attacks, which has been made popular through the Spamhaus-Cyberbunker attack incident. In this incident, CyberBunker took the advantage of open DNS resolvers to launch DNS amplification attacks, causing Spamhaus to be unreachable at times. DNS amplification and reflection attacks are typically sent to DNS servers as legitimate DNS request, in hope to receive large data size responses. The huge data size responses will eventually use up all the available bandwidth causing congestion to genuine DNS queries and responses. As such, DNS query rate limiting mechanism and higher QPS performance will not be able to counter the attack since the attacks typically come in small numbers of DNS requests. One of the ways to limit such attacks is to filter the request based on query type. Typically, DNS amplification and reflection attacks will request for ‘TXT’ or ‘ANY’ Query Type which tends to return responses with significant data size. By applying bandwidth rate limit to these query type request and large-data-size query responses, we will be able to prevent bandwidth congestion caused by these attacks. Worried about the complexity of the bandwidth rate limiting solution? Well, it only takes less than 10 lines of iRules (shown as below) on F5 DNS platform to get this enforced and implemented. when DNS_REQUEST { if { ([DNS::question type] eq "TXT") } { rateclass dns_rate_shape } } when DNS_RESPONSE { if { ([DNS::len] value > 512) } { rateclass dns_rate_shape } } Diagram 1: DNS Reflection attacks blocking genuine users from accessing LDNS server. Cache Poisoning Attacks DNSSEC is poised as the eventual and ultimate solution to counter DNS cache poisoning attacks. Though the adoption rate of DNSSEC is encouraging, it takes all parties to deploy DNSSEC signing and validation to fully protect against cache poisoning. While waiting for DNSSEC adoption rate to mature, is there any interim solution to reduce or prevent cache poisoning attacks? Based on DNS RFC standards, name servers are required to treat domain names request with case-insensitivity. In other words, the names www.foo.com and WWW.FOO.COM should resolve to the same IP address. However, most name servers will preserve the original case when echoing back the domain name in the response. Hence, by randomly varying the case of characters in domain names queried, we will be able to add entropy to requests. With this verification mechanism, the name server response must match the exact upper and lower case of every character in the name string; for instance, wWw.f5.CoM or WwW.f5.COm, which significantly reduces the success rate of cache poisoning attacks. With F5’s DNS solution, this mechanism can be enabled with just a check box on the management pane. The packet capture of the query case randomization process by F5 DNS is shown as below. As depicted in the diagram, for queries to www.google.com, F5 Cache DNS will randomize the character case of the query prior sending the query to Google’s authoritative DNS server. This greatly reduces the chances of unsolicited queries matching the domain name and DNS request transaction ID, which causes the poisoning of cached DNS records. Diagram 2: Character case randomizer in F5 DNS solution dramatically reduces the possibilities of DNS cache poisoning attacks DNS is among the hoariest of internet services that is still widely used today. Its usage continues to grow due to its simplicity and proliferation of smart devices. Hence, it is truly important that proper solution design and architecture approach are being put in place to protect the infrastructure. After all, the protection investment might be only a fraction of what you are paying for during an attack.v11.1: DNS Blackhole with iRules
Back in October, I attended a Security B-Sides event in Jefferson City (review here). One of the presenters (@bethayoung) talked about poisoning the internal DNS intentionally for known purveyors of all things bad. I indicated in my write-up that I’d be detailing an F5-based solution, and whereas a few weeks has turned into a couple months, well, here we are. As much as I had hoped to get it all together on my own, F5er Hugh O’Donnell beat me to it, and did a fantastic job. F5er Lee Orrick also contributed to the solution and I’ll have more from him in a future article. Conceptual Overview Before jumping into the nuts and bolts, I’d like to describe the solution. First, consider normal operation: Joe Anonymous is surfing and hits a popular page that has been compromised. He hits a link for a cute video about puppies and rainbows and NOT SO FAST MY FRIEND! Instead of said cute puppies and rainbows video, he ends up with a nasty case of malware and his friendly neighborhood IT staff gets to spend some time remediating the damage—if it’s caught at all. See, DNS is if not the backbone of the internet, at least several of the vertebrae. And it does its job very well. Asked and answered. Done. If you hit a link with a malicious domain, there’s a very very good chance your DNS server will have no safeguards in place, it’ll answer away. This is what a blackhole DNS solution is configured to overcome. The networking folks in the audience will be familiar with blackhole routing, and this is really no different a concept. When a user makes a query, the service inspects the destination, and if it matches a list of well known badness, it returns an address of an internal site where remediation or at least notification can take place. In either event, the request is not hitting the malicious destination, which protects user and organization. See Figure 1 for the flow detail. Building the Datagroup As with iFiles in v11.1, datagroups can also be imported via the GUI and then referenced similarly. To import your blacklisted domains (there’s a big list here: mirror1.malwaredomains.com), make sure your text editor is set for line feed terminator only (CR-LF won’t work) and use this format for each entry: “.abbcp.cn” := “harmful”, “.3dglases-panasonic-tv.com” := “zeusv2”, The first field is the domain, and the second field is a type description. The first will match your traffic, the second is strictly for classification purposes and can be edited as necessary. Intercepting the DNS Requests This solution can be implemented with LTM or GTM, though if the latter, the iRule will still need to be attached to the virtual server associated with the wideIP instead of the wideIP itself. In this article, I’ll implement the LTM-based solution. As I’ll be utilizing the new DNS:: commands, a DNS profile will need to be attached to the virtual server as well as the iRule below. Note that the blackhole class (named appropriately Blackhole_Class in the iRule below) should be present on the system for this solution to work. # Author: Hugh O'Donnell, F5 Consulting when RULE_INIT { # Set IPV4 address that is returned for Blackhole matches for A records set static::blackhole_reply_IPV4 "10.10.20.50" # Set IPV6 address that is returned for Blackhole matches for AAAA records set static::blackhole_reply_IPV6 "2001:19b8:101:2::f5f5:1d" # Set TTL used for all Blackhole replies set static::blackhole_ttl "300" } when DNS_REQUEST { # debugging statement see all questions and request details # log -noname local0. "Client: [IP::client_addr] Question:[DNS::question name] Type:[DNS::question type] Class:[DNS::question class] Origin:[DNS::origin]" # Blackhole_Match is used to track when a Query matches the blackhole list # Ensure it is always set to 0 or false at beginning of the DNS request set Blackhole_Match 0 # Blackhole_Type is used to track why this FQDN was added to the Blackhole_Class set Blackhole_Type "" # When the FQDN from the DNS Query is checked against the Blackhole class, the FQDN must start with a # period. This ensures we match a FQDN and all names to the left of it. This prevents against # malware that dynamically prepends characters to the domain name in order to bypass exact matches if {!([DNS::question name] == ".")} { set fqdn_name .[DNS::question name] } if { [class match $fqdn_name ends_with Blackhole_Class] } { # Client made a DNS request for a Blackhole site. set Blackhole_Match 1 set Blackhole_Type [class match -value $fqdn_name ends_with Blackhole_Class ] # Prevent processing by GTM, DNS Express, BIND and GTM Listener's pool. # Want to ensure we don't request a prohibited site and allow their server to identify or track the GTM source IP. DNS::return } } when DNS_RESPONSE { # debugging statement to see all questions and request details # log -noname local0. "Request: $fqdn_name Answer: [DNS::answer] Origin:[DNS::origin] Status: [DNS::header rcode] Flags: RD [DNS::header rd] RA [DNS::header ra]" if { $Blackhole_Match } { # This DNS request was for a Blackhole FQDN. Take different actions based on the request type. switch [DNS::question type] { "A" { # Clear out any DNS responses and insert the custom response. RA header = recursive answer DNS::answer clear DNS::answer insert "[DNS::question name]. $static::blackhole_ttl [DNS::question class] [DNS::question type] $static::blackhole_reply_IPV4" DNS::header ra "1" # log example: Apr 3 14:54:23 local/tmm info tmm[4694]: # Blackhole: 10.1.1.148#4902 requested foo.com query type: A class IN A-response: 10.1.1.60 log -noname local0. "Blackhole: [IP::client_addr]#[UDP::client_port] requested [DNS::question name] query type: [DNS::question type] class [DNS::question class] A-response: $static::blackhole_reply_IPV4 BH type: $Blackhole_Type" } "AAAA" { # Clear out any DNS responses and insert the custom response. RA header = recursive answer DNS::answer clear DNS::answer insert "[DNS::question name]. $static::blackhole_ttl km[DNS::question class] [DNS::question type] $static::blackhole_reply_IPV6" DNS::header ra "1" # log example: Apr 3 14:54:23 local/tmm info tmm[4694]: # Blackhole: 10.1.1.148#4902 requested foo.com query type: A class IN AAAA-response: 2001:19b8:101:2::f5f5:1d log -noname local0. "Blackhole: [IP::client_addr]#[UDP::client_port] requested [DNS::question name] query type: [DNS::question type] class [DNS::question class] AAAA-response: $static::blackhole_reply_IPV6 BH type: $Blackhole_Type" } default { # For other record types, e.g. MX, NS, TXT, etc, provide a blank NOERROR response DNS::last_act reject # log example: Apr 3 14:54:23 local/tmm info tmm[4694]: # Blackhole: 10.1.1.148#4902 requested foo.com query type: A class IN unable to respond log -noname local0. "Blackhole: [IP::client_addr]#[UDP::client_port] requested [DNS::question name] query type: [DNS::question type] class [DNS::question class] unable to respond BH type: $Blackhole_Type" } } } } This iRule handles the DNS request, responding on behalf of GTM or any DNS servers being load balanced by LTM. And since we’re handling the blackhole site, we can serve that up as well from an iRule on an HTTP virtual server. Serving the Remediation Page The remediation page can be as simple as a text message indicating malware, or it can be a little more complex to show the category of the problem site as well as provide some contact information. The iRule below is an example of the latter. # Author: Hugh O’Donnell, F5 Consulting when HTTP_REQUEST { # the static HTML pages include the logo that is referenced in HTML as corp-logo.gif # intercept requests for this and reply with the image that is stored in an iFile defined in RULE_INIT below if {[HTTP::uri] ends_with "/_maintenance-page/corp-logo.png" } { # Present HTTP::respond 200 content $static::corp_logo } else { # Request for Blackhole webpage. Identify what type of block was in place switch -glob [class match -value ".[HTTP::host]" ends_with Blackhole_Class ] { "virus" { set block_reason "Virus site" } "phishing" { set block_reason "Phishing site" } "generic" { set block_reason "Unacceptable Usage" } default { set block_reason "Denied Per Policy - Other Sites" } } # Log details about the blackhole request to the remote syslog server log -noname local0. "Blackhole: From [IP::client_addr]:[TCP::client_port] \ to [IP::local_addr]:[TCP::local_port], [HTTP::request_num], \ [HTTP::method],[HTTP::uri],[HTTP::version], [HTTP::host], [HTTP::header value Referer], \ [HTTP::header User-Agent], [HTTP::header names],[HTTP::cookie names], BH category: $block_reason," # Send an HTML page to the user. The page is defined in the RULE_INIT event below HTTP::respond 200 content "$static::block_page [HTTP::host][HTTP::uri] $static::after_url $block_reason $static::after_block_reason " } } when RULE_INIT { # load the logo that was stored as an iFile set static::corp_logo [ifile get "/Common/f5ball"] # Beginning of the block page set static::block_page " <html lang=\"en_US\"> <head> <title>Web Access Denied - Enterprise Network Operations Center</title> <meta http-equiv=\"Content-Type\" content=\"text/html; charset=us-ascii\"> <meta http-equiv=\"CACHE-CONTROL\" content=\"NO-CACHE\"> <meta http-equiv=\"PRAGMA\" content=\"NO-CACHE\"> <meta http-equiv=\"EXPIRES\" content=\"Mon, 22 Jul 2002 11:12:01 GMT\"> <style> <!-- .mainbody { background-color: #C0C0C0; color: #000000; font-family: Verdana, Geneva, sans-serif; font-size: 12px; margin: 0px; padding: 20px 0px 20px 0px; position: relative; text-align: center; width: 100%; } .bdywrpr { width:996px; height:auto; text-align:left; margin:0 auto; z-index:1; position: relative; } #banner-wrapper { width: 950px; padding: 0px; margin: 0px; overflow:hidden; background-color: #FFFFFF; background-repeat: no-repeat; } #banner-image { float: left; margin-left: auto; margin-right: auto; padding: 3px 0px 2px 7px; width: 950px; } #textbody { background-color: #FFFFFF; color: #000000; font-family: Verdana, Geneva, sans-serif; font-size: 13px; width: 950px; padding:0px; text-align:justify; margin: 0px; } --> </style> </head> <body class=\"mainbody\"> <div class=\"bdywrpr\"> <div id=\"banner-wrapper\"> <!-- BANNER --> <div id=\"banner-image\"> <center><img src=\"/_maintenance-page/corp-logo.png\" alt=\"Enterprise Network Operations Center\"></center> </div> </div> <div id=\"textbody\"> <table border=\"0\" cellpadding=\"40\"><tr><td> <center><p style=\"font-size:18px\"><b>Access has been denied.<br><br> URL: " set static::after_url "</p></center></b></font> <br> Your request was denied because it is blacklisted in DNS. <br><br> Blacklist category: " set static::after_block_reason "<br> <p> The Internet Gateways are for official use only. Misuse violates policy. If you believe that this site is categorized incorrectly, and that you have a valid business reason for access to this site please contact your manager for approval and the Enterprise Network Operations Center via <br><br> E-mail: <a href=\"mailto:enoc@example.com\">enoc@example.com</a> <br><br> Please use the Web Access Request Form and include a business justification. Only e-mail that originates from valid internal e-mail addresses will be processed. If you do not have a valid e-mail address, your manager will need to submit a request on your behalf. </center> <p> <font size=-1><i>Generated by bigip1.f5.com.</i></font> </td></tr></table> </div> </div> </body> </html> " } Note that the remediation page references an iFile for a logo. For details on configuring iFiles, please reference my article on iFiles. Also note that in addition to the client getting a heads-up notification of malfeasance, the visit is logged so other processes, individuals can act on the information. The Results First, our DNS query and response. Rather than test out a real well-known bad site, I added espn.com to my blacklist so if I forgot a step and leaked through to the real site I wouldn’t compromise anything. The response from my DNS virtual server is shown in Figure 2 below. You can see that the address matches the address set in the iRule as our blackhole IPv4 address. Also, the log information from that DNS query: Dec 28 15:35:08 tmm info tmm[6883]: Blackhole: 10.10.20.251#57714 requested espn.com query type: A class IN A-response: 10.10.20.50 BH type: sports Next, the resulting remediation page in my browser (Figure 3): And finally, the log entry from the HTTP request: Dec 28 15:35:08 tmm info tmm[6883]: Blackhole: From 10.10.20.251:32447 to 10.10.20.50:80, 1, GET,/,1.1, espn.com, , Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.63 Safari/535.7, Host Connection User-Agent Accept Accept-Encoding Accept-Language Accept-Charset,, BH category: Denied Per Policy - Other Sites, Conclusion This is a wicked application of iRules with new DNS and file handling features delivered in v11.1. If you wanted to take it even further, you could use sideband connections and reference an external list instead of a datagroup that will need constant refreshing. The GTM version of this solution is documented in CrowdSRC. If you’re curious about the DNS commands used in the iRule above, I’ll be discussing them in my next tech tip, so check back soon! Note:For the LTMsolution presented above, the DNSServices module or the GTM module is required to be licensed.Configuring Decision Logging for the F5 BIG-IP Global Traffic Manager
I was working on a GTM solution and with my limited lab I wanted to make sure that the decisions that F5 BIG-IP Global Traffic Manager made at the wideIP and pool level were as evident in the logs as they were consistent in my test results. It turns out there are some fancy little checkboxes in the wideIP configuration that you can check to enable such logs. You might notice, however, that upon enabling these checkboxes the logs are nowhere to be found. This is because there are other necessary steps. You need to configure a few objects to get those logs flowing. Log Publisher The first object is the log publisher. For as much detail as flows in the decision logging, I’d highly recommend using an HSL profile to log to a remote server, but for the purposes of testing I used the local syslog. This can also be done with tmsh. sys log-config publisher gtm_decision_logging { destinations { local-syslog { } } } DNS Logging Profile Next, create a DNS logging profile, make sure to select the Log Publisher you created in the previous step. For testing purpose I enabled the log responses and query ID as well, but those are disabled by default. This also can be created in tmsh. ltm profile dns-logging gtm_decision_logging { enable-response-logging yes include-query-id yes log-publisher gtm_decision_logging } Custom DNS Profile Now create a custom DNS profile. The only custom properties necessary are at the bottom of the profile where you enable logging and select the logging profile. This can also be configured in tmsh. ltm profile dns gtm_decision_logging { app-service none defaults-from dns enable-logging yes log-profile gtm_decision_logging } Apply the DNS Profile Now that all the objects are created, you can reference the DNS profile in the listener. in tmsh, you can modify the listener by adding the profile or if one already exists, replacing it. modify gtm listener gtmlistener profiles replace-all-with { udp_gtm_dns gtm_decision_logging } Log Details Once you have all the objects configured and the DNS profile referenced in your listener, the logging should be hitting /var/log/ltm now. For this first query, the emea pool is selected, but there is no probe data for my primary load balancing method, and the none alternate method skips to the fallback, which uses the configured fallback IP to respond to the client. 2015-06-03 08:54:21 ltm1.dc.test qid 11139 from 192.168.102.1#64536: view none: query: my.example.com IN A + (192.168.102.5%0) 2015-06-03 08:54:21 ltm1.dc.test qid 11139 from 192.168.102.1#64536 [my.example.com A] [round robin selected pool (emea)] [pool member check succeeded (vip3:192.168.103.12) - pool member state is available (green)] [QoS skipped pool member (vip3:192.168.103.12) - path has unmeasured RTT] [pool member check succeeded (vip4:192.168.103.13) - pool member state is available (green)] [QoS skipped pool member (vip4:192.168.103.13) - path has unmeasured RTT] [failed to select pool member by preferred load balancing method] [Using none load balancing method] [failed to select pool member by alternate load balancing method] [selected configured fallback IP] 2015-06-03 08:54:21 ltm1.dc.test qid 11139 to 192.168.102.1#64536: [NOERROR qr,aa,rd] response: my.example.com. 30 IN A 192.168.103.99; In this second request, the emea pool is again selected, but now there is probe data, so the pool member is selected as appropriate. 2015-06-03 08:55:43 ltm1.dc.test qid 6201 from 192.168.102.1#61503: view none: query: my.example.com IN A + (192.168.102.5%0) 2015-06-03 08:55:43 ltm1.dc.test qid 6201 from 192.168.102.1#61503 [my.example.com A] [round robin selected pool (emea)] [pool member check succeeded (vip3:192.168.103.12) - pool member state is available (green)] [QoS selected pool member (vip3:192.168.103.12) - QoS score (2082756232) is higher] [pool member check succeeded (vip4:192.168.103.13) - pool member state is available (green)] [QoS skipped pool member (vip4:192.168.103.13) from two pool members with equal scores] [QoS selected pool member (vip3:192.168.103.12)] 2015-06-03 08:55:43 ltm1.dc.test qid 6201 to 192.168.102.1#61503: [NOERROR qr,aa,rd] response: my.example.com. 30 IN A 192.168.103.12; In this final request, the americas pool is selected, but there is no valid topology score for the pool members, so query is refused. 2015-06-03 08:55:53 ltm1.dc.test qid 23580 from 192.168.102.1#59437: view none: query: my.example.com IN A + (192.168.102.5%0) 2015-06-03 08:55:53 ltm1.dc.test qid 23580 from 192.168.102.1#59437 [my.example.com A] [round robin selected pool (americas)] [pool member check succeeded (vip1:192.168.103.10) - pool member state is available (green)] [QoS selected pool member (vip1:192.168.103.10) - QoS score (0) is higher] [pool member check succeeded (vip2:192.168.103.11) - pool member state is available (green)] [QoS skipped pool member (vip2:192.168.103.11) from two pool members with equal scores] [QoS selected pool member (vip1:192.168.103.10)] [topology load balancing method failed to select pool member (vip1:192.168.103.10) - topology score is 0] [failed to select pool member by preferred load balancing method] [selected configured option Return To DNS] 2015-06-03 08:55:53 ltm1.dc.test qid 23580 to 192.168.102.1#59437: [REFUSED qr,rd] response: empty Yeah, yeah, skip all that and give me the good stuff If you want to test it quickly, you can save the config below to a file (/var/tmp/gtmlogging.txt in this example) and then merge it in. Finally, modify the wideIP and listener and you’re good to go! ### ### configuration: /var/tmp/gtmlogging.txt ### sys log-config publisher gtm_decision_logging { destinations { local-syslog { } } } ltm profile dns-logging gtm_decision_logging { enable-response-logging yes include-query-id yes log-publisher gtm_decision_logging } ltm profile dns gtm_decision_logging { app-service none defaults-from dns enable-logging yes log-profile gtm_decision_logging } ### ### Merge Command ### tmsh load sys config merge file /var/tmp/gtmlogging.txt ### ### Modify wideIP and Listener ### tmsh modify gtm wideip my.example.com load-balancing-decision-log-verbosity { pool-member-selection pool-member-traversal pool-selection pool-traversal } tmsh modify gtm listener gtmlistener profiles replace-all-with { udp_gtm_dns gtm_decision_logging } tmsh save sys configDNSSEC: Is Your Infrastructure Ready?
A few months ago, we teamed with Infoblox for a DNSSEC webinar. Jonathan George, F5 Product Marketing Manager, leads with myself and Cricket Liu of Infoblox as background noise. He’s a blast as always and certainly knows his DNS. So, learn how F5 enables you to deploy DNSSEC quickly and easily into an existing GSLB environment with BIG-IP Global Traffic Manager (GTM). BIG-IP GTM streamlines encryption key generation and distribution by dynamically signing DNS responses in real-time. Running time: 49:20 &lt;/p&gt; &lt;p&gt;ps&lt;/p&gt; &lt;p&gt;Resources:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href=&quot;http://www.f5.com/news-press-events/web-media/&quot; _fcksavedurl=&quot;http://www.f5.com/news-press-events/web-media/&quot;&gt;F5 Web Media&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;http://www.youtube.com/user/f5networksinc&quot; _fcksavedurl=&quot;http://www.youtube.com/user/f5networksinc&quot;&gt;F5 YouTube Channel&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;http://www.f5.com/products/big-ip/global-traffic-manager.html&quot; _fcksavedurl=&quot;http://www.f5.com/products/big-ip/global-traffic-manager.html&quot;&gt;BIG-IP GTM&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;http://www.f5.com/pdf/white-papers/dnssec-wp.pdf&quot; _fcksavedurl=&quot;http://www.f5.com/pdf/white-papers/dnssec-wp.pdf&quot;&gt;DNSSEC: The Antidote to DNS Cache Poisoning and Other DNS Attacks (whitepaper)&lt;/a&gt; | &lt;a href=&quot;http://devcentral.f5.com/s/weblogs/interviews/archive/2009/12/04/audio-tech-brief-dnssec-the-antidote-to-dns.aspx&quot; _fcksavedurl=&quot;http://devcentral.f5.com/s/weblogs/interviews/archive/2009/12/04/audio-tech-brief-dnssec-the-antidote-to-dns.aspx&quot;&gt;Audio&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;http://www.cricketondns.com&quot; _fcksavedurl=&quot;http://www.cricketondns.com&quot;&gt;Cricket on DNS&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href=&quot;http://www.youtube.com/user/InfobloxInc&quot; _fcksavedurl=&quot;http://www.youtube.com/user/InfobloxInc&quot;&gt;Infoblox YouTube Channel&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Technorati Tags: &lt;a href=&quot;http://devcentral.f5.com/s/weblogs/psilva/psilva/psilva/archive/2011/05/09/&quot; _fcksavedurl=&quot;http://devcentral.f5.com/s/weblogs/psilva/psilva/psilva/archive/2011/05/09/&quot;&gt;F5&lt;/a&gt;, &lt;a href=&quot;http://technorati.com/tags/webinar&quot; _fcksavedurl=&quot;http://technorati.com/tags/webinar&quot;&gt;webinar&lt;/a&gt;, &lt;a href=&quot;http://technorati.com/tags/Pete+Silva&quot; _fcksavedurl=&quot;http://technorati.com/tags/Pete+Silva&quot;&gt;Pete Silva&lt;/a&gt;, &lt;a href=&quot;http://technorati.com/tags/security&quot; _fcksavedurl=&quot;http://technorati.com/tags/security&quot;&gt;security&lt;/a&gt;, &lt;a href=&quot;http://technorati.com/tag/business&quot; _fcksavedurl=&quot;http://technorati.com/tag/business&quot;&gt;business&lt;/a&gt;, &lt;a href=&quot;http://technorati.com/tag/education&quot; _fcksavedurl=&quot;http://technorati.com/tag/education&quot;&gt;education&lt;/a&gt;, &lt;a href=&quot;http://technorati.com/tag/technology&quot; _fcksavedurl=&quot;http://technorati.com/tag/technology&quot;&gt;technology&lt;/a&gt;, &lt;a href=&quot;http://technorati.com/tags/internet&quot; _fcksavedurl=&quot;http://technorati.com/tags/internet&quot;&gt;internet, &lt;/a&gt;&lt;a href=&quot;http://technorati.com/tags/big-ip&quot; _fcksavedurl=&quot;http://technorati.com/tags/big-ip&quot;&gt;big-ip&lt;/a&gt;, &lt;a href=&quot;http://technorati.com/tag/dnssec&quot; _fcksavedurl=&quot;http://technorati.com/tag/dnssec&quot;&gt;dnssec&lt;/a&gt;, &lt;a href=&quot;http://technorati.com/tags/infoblox&quot; _fcksavedurl=&quot;http://technorati.com/tags/infoblox&quot;&gt;infoblox&lt;/a&gt; &lt;a href=&quot;http://technorati.com/tags/dns&quot; _fcksavedurl=&quot;http://technorati.com/tags/dns&quot;&gt;dns&lt;/a&gt;&lt;/p&gt; &lt;table border=&quot;0&quot; cellspacing=&quot;0&quot; cellpadding=&quot;2&quot; width=&quot;378&quot;&gt;&lt;tbody&gt; &lt;tr&gt; &lt;td valign=&quot;top&quot; width=&quot;200&quot;&gt;Connect with Peter: &lt;/td&gt; &lt;td valign=&quot;top&quot; width=&quot;176&quot;&gt;Connect with F5: &lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;td valign=&quot;top&quot; width=&quot;200&quot;&gt;&lt;a href=&quot;http://www.linkedin.com/pub/peter-silva/0/412/77a&quot; _fcksavedurl=&quot;http://www.linkedin.com/pub/peter-silva/0/412/77a&quot;&gt;&lt;img style=&quot;border-bottom: 0px; border-left: 0px; display: inline; border-top: 0px; border-right: 0px&quot; title=&quot;o_linkedin[1]&quot; border=&quot;0&quot; alt=&quot;o_linkedin[1]&quot; src=&quot;http://devcentral.f5.com/s/weblogs/images/devcentral_f5_com/weblogs/macvittie/1086440/o_linkedin.png&quot; _fcksavedurl=&quot;http://devcentral.f5.com/s/weblogs/images/devcentral_f5_com/weblogs/macvittie/1086440/o_linkedin.png&quot; width=&quot;24&quot; height=&quot;24&quot; /&gt;&lt;/a&gt; &lt;a href=&quot;http://devcentral.f5.com/s/weblogs/psilva/Rss.aspx&quot; _fcksavedurl=&quot;http://devcentral.f5.com/s/weblogs/psilva/Rss.aspx&quot;&gt;&lt;img style=&quot;border-bottom: 0px; border-left: 0px; display: inline; border-top: 0px; border-right: 0px&quot; title=&quot;o_rss[1]&quot; border=&quot;0&quot; alt=&quot;o_rss[1]&quot; src=&quot;http://devcentral.f5.com/s/weblogs/images/devcentral_f5_com/weblogs/macvittie/1086440/o_rss.png&quot; _fcksavedurl=&quot;http://devcentral.f5.com/s/weblogs/images/devcentral_f5_com/weblogs/macvittie/1086440/o_rss.png&quot; width=&quot;24&quot; height=&quot;24&quot; /&gt;&lt;/a&gt; &lt;a href=&quot;http://www.facebook.com/f5networksinc&quot; _fcksavedurl=&quot;http://www.facebook.com/f5networksinc&quot;&gt;&lt;img style=&quot;border-bottom: 0px; border-left: 0px; display: inline; border-top: 0px; border-right: 0px&quot; title=&quot;o_facebook[1]&quot; border=&quot;0&quot; alt=&quot;o_facebook[1]&quot; src=&quot;http://devcentral.f5.com/s/weblogs/images/devcentral_f5_com/weblogs/macvittie/1086440/o_facebook.png&quot; _fcksavedurl=&quot;http://devcentral.f5.com/s/weblogs/images/devcentral_f5_com/weblogs/macvittie/1086440/o_facebook.png&quot; width=&quot;24&quot; height=&quot;24&quot; /&gt;&lt;/a&gt; &lt;a href=&quot;http://twitter.com/psilvas&quot; _fcksavedurl=&quot;http://twitter.com/psilvas&quot;&gt;&lt;img style=&quot;border-bottom: 0px; border-left: 0px; display: inline; border-top: 0px; border-right: 0px&quot; title=&quot;o_twitter[1]&quot; border=&quot;0&quot; alt=&quot;o_twitter[1]&quot; src=&quot;http://devcentral.f5.com/s/weblogs/images/devcentral_f5_com/weblogs/macvittie/1086440/o_twitter.png&quot; _fcksavedurl=&quot;http://devcentral.f5.com/s/weblogs/images/devcentral_f5_com/weblogs/macvittie/1086440/o_twitter.png&quot; width=&quot;24&quot; height=&quot;24&quot; /&gt;&lt;/a&gt; &lt;/td&gt; &lt;td valign=&quot;top&quot; width=&quot;176&quot;&gt; &lt;a href=&quot;http://bitly.com/nIsT1z?r=bb&quot; _fcksavedurl=&quot;http://bitly.com/nIsT1z?r=bb&quot;&gt;&lt;img style=&quot;border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px&quot; title=&quot;o_facebook[1]&quot; border=&quot;0&quot; alt=&quot;o_facebook[1]&quot; src=&quot;http://devcentral.f5.com/s/weblogs/images/devcentral_f5_com/weblogs/macvittie/1086440/o_facebook.png&quot; _fcksavedurl=&quot;http://devcentral.f5.com/s/weblogs/images/devcentral_f5_com/weblogs/macvittie/1086440/o_facebook.png&quot; width=&quot;24&quot; height=&quot;24&quot; /&gt;&lt;/a&gt; &lt;a href=&quot;http://bitly.com/rrAfiR?r=bb&quot; _fcksavedurl=&quot;http://bitly.com/rrAfiR?r=bb&quot;&gt;&lt;img style=&quot;border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px&quot; title=&quot;o_twitter[1]&quot; border=&quot;0&quot; alt=&quot;o_twitter[1]&quot; src=&quot;http://devcentral.f5.com/s/weblogs/images/devcentral_f5_com/weblogs/macvittie/1086440/o_twitter.png&quot; _fcksavedurl=&quot;http://devcentral.f5.com/s/weblogs/images/devcentral_f5_com/weblogs/macvittie/1086440/o_twitter.png&quot; width=&quot;24&quot; height=&quot;24&quot; /&gt;&lt;/a&gt; &lt;a href=&quot;http://bitly.com/neO7Pm?r=bb&quot; _fcksavedurl=&quot;http://bitly.com/neO7Pm?r=bb&quot;&gt;&lt;img style=&quot;border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px&quot; title=&quot;o_slideshare[1]&quot; border=&quot;0&quot; alt=&quot;o_slideshare[1]&quot; src=&quot;http://devcentral.f5.com/s/weblogs/images/devcentral_f5_com/weblogs/macvittie/1086440/o_slideshare.png&quot; _fcksavedurl=&quot;http://devcentral.f5.com/s/weblogs/images/devcentral_f5_com/weblogs/macvittie/1086440/o_slideshare.png&quot; width=&quot;24&quot; height=&quot;24&quot; /&gt;&lt;/a&gt; &lt;a href=&quot;http://bitly.com/mOVxf3?r=bb&quot; _fcksavedurl=&quot;http://bitly.com/mOVxf3?r=bb&quot;&gt;&lt;img style=&quot;border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px&quot; title=&quot;o_youtube[1]&quot; border=&quot;0&quot; alt=&quot;o_youtube[1]&quot; src=&quot;http://devcentral.f5.com/s/weblogs/images/devcentral_f5_com/weblogs/macvittie/1086440/o_youtube.png&quot; _fcksavedurl=&quot;http://devcentral.f5.com/s/weblogs/images/devcentral_f5_com/weblogs/macvittie/1086440/o_youtube.png&quot; width=&quot;24&quot; height=&quot;24&quot; /&gt;&lt;/a&gt;&lt;/td&gt; &lt;/tr&gt; &lt;/tbody&gt;&lt;/table&gt;&lt;/body&gt;&lt;/html&gt; ps Resources: F5 Web Media F5 YouTube Channel BIG-IP GTM DNSSEC: The Antidote to DNS Cache Poisoning and Other DNS Attacks (whitepaper) | Audio Cricket on DNS Infoblox YouTube Channel
