Many firewalls and IPS solutions will lay claim to DNS application layer gateways (ALGs), but these solutions do not address many of the more modern threats to DNS infrastructure. These threats include DNS DoS attacks, reflection attacks, tunneling, and cache poisoning, to name but a few. Since DNS is largely UDP-based, traditional mechanisms for mitigation based on TCP aren't effective, and source-IP limiting or denylisting is mostly useless since source address spoofing is trivial with UDP for transport.
What these DNS attack vectors require is a highly scalable solution able to inspect the entire DNS query for deeper markers of either good or bad behavior.
When it comes to protecting Internet-facing DNS from outside threats, even DNS servers acting as a hidden controller to a cloud DNS provider, the first priority is scale. Most DNS infrastructures cannot keep pace with millions of requests per second, and in a DNS reflection attack where requests are often for legitimate records, there is very little facility to differentiate good traffic from bad. In this instance, the available ISP bandwidth may not be exceeded, but the query volume has overwhelmed the DNS server infrastructure's ability to respond.
Even a mid-range BIG-IP DNS platform can serve up 1.3 million query responses per second (RPS), as an authoritative screen for your DNS server environment. BIG-IP DNS (formerly GTM) has a proprietary DNS server called DNS Express which enables this massive performance while completely proxying all DNS requests for your DNS server environment. DNS Express isn't BIND, and as such has not shared the same vulnerability surface as the world's most ubiquitous DNS server software. A pair of BIG-IP DNS appliances could easily serve over 2M DNS RPS, while reducing the capacity requirements of the back-end DNS server infrastructure.
Adding BIG-IP Advanced Firewall Manager (AFM) enables a range of capabilities for DNS DoS detection and DNS protocol validation to the platform. BIG-IP AFM can detect surges in DNS traffic (which will be handled by DNS Express, if BIG-IP DNS is enabled). DNS protocol validation ensures that DNS requests are not malformed before they are even handed off to DNS Express. Via the DNS protocol security policy, it's also possible to ensure that only supported record types and opcodes. In other words, answer only A record requests and not TXT records, which are a common vector for DoS attacks and DNS tunneling. DNS Express will clean up any NXDOMAIN (non-existent hostname) requests by rejecting those, as the authoritative DNS screen.
Completing the picture of a full-featured, application layer DNS firewall are the BIG-IP DNS features such as DNSSEC signing (providing cache poisoning protection), DNS64 translation (improving IPv6 readiness and AAAA record responses), and good old global server load-balancing (GSLB).
An overlooked part of the DNS firewall and security challenge is the outbound DNS traffic.
Malware - once it has infected/compromised a machine on your network - typically seeks to call C&C (command and control) servers. The malware may be seeking instructions, additional files to complete its installation, or sending stolen data to a drop zone, among other possibilities. The challenge for this malware is finding a path out of the network that isn't blocked or monitored. This is the getaway part of the data-heist.
Every corporate network, whether in the data center, or the campus LAN, has a few open ports so that the servers or employee desktops can function. Layer 4 ports 80 (HTTP), 443 (HTTPS), and DNS (53) are probably the three most common. Malware will accordingly be programmed to seek a path out over these ports. DNS will be popular among the malware developers because most networks have some kind of web content filter or gateway (such as a web proxy or BIG-IP SWG), and are well-inspected paths. DNS, on the other hand, is much more difficult to secure. Most outbound DNS architectures are designed to handle the scale of every server or workstation making DNS lookup requests, so you see features like caching resolvers on DNS servers and on BIG-IP designed to cache lookups and act as an aggregator for requests to limit impact on network traffic.
These DNS caching resolvers usually have an open path to the Internet over port 53, since these servers a trusted source. An anti-malware or IPS/IDS solution might pick up some anomalous DNS traffic patterns sourced from that caching resolver, but the digital forensics/incident response (DFIR) team will be left grasping straws trying to trace the anomalous traffic back to its origin since caching resolvers are built for performance not security. In other words, caching resolvers provide limited, if any, logging. BIG-IP DNS is one of few solutions offering per-request and per-response logging of DNS traffic, in addition to a high-performance DNS caching resolver.
In addition, the DNS proxy in BIG-IP offers protocol validation functions (ensuring that DNS requests and responses are properly formatted and within prescribed size limits), record type filters (ensuring that only allowed records can be requested), and inspection capabilities (ensuring that data isn't being smuggled out via DNS requests). The last vector is commonly called DNS tunneling, and will use request types such as TXT records which enables the malware to embed more data. For more advanced DNS tunneling vectors, the extensibility of iRules is available for the DNS protocol, as well.
In summary, BIG-IP DNS combined with AFM provides a wealth of inspection, mitigation, and logging facilities to augment what may be found via third-party IPS or malware detection facilities.