Who needs a Bot Army these days?

It’s been a while - a long while - since I last (officially) blogged. Too many distractions with a new role, new travels, and a new family member - sucks away the creative juices. Alas, sitting at an airport lounge after a 3 city Security Roadshow in Bombay/Delhi/Bangalore brings out the adrenaline rush of blogging once more.

I’ve been evangelizing the need for DoS protection for quite a while. Events of last year (2013), and even the beginning of 2014, have made my job easier (Happy New Year!). I used to equate DoS (rather Distributed DoS) attacks with Bot Armies - kinda like the Orcs that we saw in the LOTR (Lord of the Rings**) saga. The Bots (Orcs) still exist - but seems like the Bot Herder (I’ll call him “Sauron” **) is summoning a new type of Army these days - a new generation of orcs used to carry out his mission to wipe a particular service/server from the face of the Internet world - You and Me!

Yes - US!

Who needs a Bot Army these days? There are gazillions of Internet users and servers out there. There isn’t a need for any kind of endpoint infection/malware. That would be troublesome. All you need is an unsuspecting (and poorly protected) + popular web server, and some programming flair with JavaScript (JS) or iFrames. Then you need to compromise that popular web server and input the malicious JS/iFrame into the landing web page. So when a user visits this compromised popular website (e.g. an online gossip portal) to read about the latest Hollywood gossips, the hidden JavaScripts or iFrames gets executed within his browser and causes him to send multiple GET requests to the victim. And the victim gets hoarded with a torrent of GET requests, and if its not sufficiently and appropriately protected, guess what happens? Bam!

2014 began with an interesting new type of attack against a couple of gaming sites like League of Legends, Steam and Battle.net using yet another type of “botnet”, except that these weren’t bots but actual Network Time Protocol (NTP) servers. The NTP protocol is used by many servers and endpoints to sync system clocks - for example my mac uses time.asia.apple.com. Leveraging the inherent “trust” that the UDP protocol exhibits (aka stateless), and a relatively forgotten command called “monlist”, this is how it works. From my laptop, a small query of 234bytes returns a response of 100 packets each of 482bytes, producing an amplification factor of approximately 205x (you get the hint). So I need to be armed with a list of open and unpatched NTP servers (which isn’t difficult to find - there are automated programs out there that can “assist"), and spoof the IP address of someone whom I don’t like, and shazam.

An example of the now infamous “monlist” command for an unpatched NTP server looks like:

This reminds me of the DNS Reflective Amplification attacks during the Cyberbunker-Spamhaus saga in March of last year (2013).

Again, the trust model of the DNS protocol was exploited, and what that didn’t help with the problem were those 20-something million open DNS resolvers which neither perform any kind of filtering nor any request verification or traffic management of DNS responses. And as a result, 300Gbps of DNS responses towards an unsuspecting victim, which even caused collateral damage to International Peering exchanges.

Now, how do we deal with these?

For the DNS and NTP reflective amplification attacks, apart from deciding to not be an open DNS resolver, or disabling the “monlist” command, if you are a service or hosting provider to one of these potential “Orcs”, one simple way is to monitor the outbound BPS/PPS/Number of connections originating from the server. When it gets anomalously high, move to the next step of monitoring perhaps the Top 10 destinations these guys are sending towards. It doesn’t make sense that a DNS server out there is sending 1M DNS “Any” responses per second to www.f5.com. Apply an appropriate rate shaping policy as needed and that will help protect your own network infrastructure, save some peering/transit costs, and help the poor victim.

The HTTP reflection attack is a bit more challenging, given that as mentioned previously - it is REAL HTTP traffic from REAL browsers out there. You can’t filter based on malformed HTTP packets. You can’t filter based on URL since it’s typically a legitimate URL. 

This calls for Anti-Bot Intelligence, like:

1. TCP SYN Cookie verification (L4)
2. HTTP Redirection verification (L7)
3. JavaScript / HTTP Cookie verification (L7)
4. CAPTCHA verification (L7)

With the ability to step-up countermeasures dynamically and when needed. Of course, the latter verification methods are higher in the OSI stack (Layer 7), more advanced and computationally more expensive to perform.

Here at F5, we’ve found that (3) and (4) have been extremely effective in terms of combating DoS tools/scripts and even the hidden JavaScripts/iFrames mentioned above. These guys typically bail out when it comes to solving complex JavaScript puzzles or checking of mouse/keyboard movements etc., and not to mention solving CAPTCHA puzzles. Once this happens, throw the errant source into a penalty box for a period of time, then repeat the rinse and lather. Dynamically increase this penalty box timeout when the errant source fails multiple times. This will effectively thwart any attempt from the errant source in sending HTTP requests to the victim server.

Lastly, work with your upstream service provider, or engage a cloud-based anti-ddos service when the attacks become too large to handle. Sound the SOS! It is common-sense the only possible way to mitigate 300Gbps floods is in the service provider cloud / cloud-based anti-ddos service, likely using a distributed scrubbing “anycast” model, where each distributed scrubbing center sucks in the traffic destined for the victim like a big washing machine and starts cleaning.

That said, there are many ways to skin a cat - likewise, there are many ways to solve a DoS problem. Pay particular attention to attacks as they move up the OSI stack toward Layer 7. These are typically harder to mitigate, but fortunately not for F5.

So if you’re thinking that botnet armies are the Sole source of menace in the DoS world out there, they’re not. Everyday Internet users like you and me could be an unsuspecting participant in the world’s largest DoS to be.

** Source: The Lord of the Rings, J.R.R Tolkien, 1954