Introduction
Five years ago Raymond Pompon wrote about DNS water torture attacks being one of the attacks we still see on the internet. At that time they were going through a resurgence, and recently we have been seeing an uptick of DNS water torture attacks on F5 customers.
Anatomy of a DNS Water Torture Attack
These attacks are not particularly sophisticated, typically the attacker generates a large number of DNS queries for nonexistant leaf domains under the target domain name. When we first started seeing these the generated random name was fairly unsophisticated, often being a fixed length and consistency. Over time they progressed to be more sophisticated, these days they appear to usually be generated from a random combination of wordlists with some assorted alphanumeric characters tossed in. This means unlike years ago when we could just configure an iRule to drop requests that fit a regex, these requests can't be filtered that way.
The intent of these randomly generated requests is to saturate the DNS server infrastructure of the target domain and cause degradation of service. Typically volumes will not be anywhere near saturating the internet pipe to that infrastructure. The use of random names and sometimes random capitalization of the DNS name being queried is intended to bypass any caching infrastructure between the attackers' bots and the domain's DNS infrastructure. These attacks often use a botnet, but in some cases they can originate from spoofed IPs when the attacker has access to an ISP that does not filter outbound packets for bogons (originating IPs not owned by the network).
Analyzing Traffic
You can capture dns requests on the BIG-IP using a command like:
tcpdump -s 0 -ni 0.0:nnn -c10000 -w /shared/tmp/capture_`date +%F-%H%M`.pcap port 53
This capture command will create a pcap file in the /shared/tmp directory with the date and time in the filename and put 10,000 packets in it before terminating. The file may be retrieved using an scp/sftp client.
Personally I use NetworkMiner to review this pcap file, but you can also do it in Wireshark/tshark or use tcpdump to print out the DNS requests:
tcpdump -n -r /shared/tmp/capture_2023-06-30-1824.pcap
DNS Water Torture requests may look like:
jonbenetramsey.www.contoso.com
b782349889.dev.contoso.com
opt-harrisson-22q13.contoso.com
xai1Ieph2tuo.www-5.contoso.com
Ofiipoh6vo-hshu.dev.contoso.com
9ic-hienim-0ni9yeih7j.water.contoso.com
Strategies
The most common strategy I recommend is to shore up the DNS infrastructure so the attack does not succeed in causing any impact. Often attackers will eventually stop the attack when its not having any impact, sometimes they will come back with more volume or increase the volume, but most attacks we have seen involve the attacker throwing all they have at the attack. In at least some cases the attacker will demand a ransom if the attack is successful at causing a degradation of service.
The next best strategy and one that should be put in place if you have AFM is to create a DoS profile with DNS protections. For those settings you will want to set A/AAAA query flood to a value that is just under the maximum your DNS implementation is able to handle, this is entirely to keep the attack from causing the DNS infrastructure to fail (run out of memory or crypto transactions per second). You can set a value or leave the default value for NXDOMAIN flood, which is the specific attack vector in this case. We are setting both here because in some cases the NXDOMAIN attack will not be detected by AFM, see the Limitations section below.
One of the major limitations in this type of attack is that it takes place across the DNS infrastructure of the internet, so blocking specific IP addresses is limited by ISPs and other networks aggregating DNS requests using their DNS resolvers. This limits our ability to use bad actor blocking in this type of attack, since A/AAAA rate limits that will catch the bad actors will also catch major ISPs and other large networks.
In some cases IP Intelligence may be effective at blocking some of the bad actor IPs, but since the requests may be going through ISP resolvers the value of IP Intelligence is limited in this case. GeoIP Blocking may also help some in cutting down the volume, but typically the botnets these attacks originate span the world and don't have any concentration in any particular country.
One of the common methods of shoring up DNS infrastructure with BIG-IP DNS is using DNS Express to serve DNS responses from a memory cache on the BIG-IP. This allows load to be taken off the DNS server and be handled by the BIG-IP. We recommend considering DNS Express when using BIND on the BIG-IP with ZoneRunner, as BIND on BIG-IP is not optimized for high volume DNS.
Limitations
If your signing DNSSEC responses using a key in a FIPS HSM you will most likely want to set the DB key dnssec.maxnsec3persec to a value that is a fraction of the transaction capacity of the FIPS HSM. For example if it can handle 4000 transactions per second, a good value may be 1000. This will limit the number of NXDOMAIN responses to 1000 per second and keep the FIPS HSM from being overloaded with transactions. Make sure your running a version of BIG-IP TMOS in which ID1076401 is fixed before changing this value.
If your using the local BIND installation to serve requests, the NXDOMAIN vector in AFM may not work, we have logged ID1186449 for this issue.
How to Contact the F5 SIRT
If you are an F5 customer, you can visit support.f5.com to open a case for security incident. We also have a number of regional tool free numbers in case you want to raise a case by telephone. When raising a case for a security emergency please provide as much information as possible, especially what products are involved, so we can triage the case to an engineer trained on those products.
If you are not an F5 customer and want to report a vulnerability, please refer to our Report a Vulnerability page for details on how to report a vulnerability.
