DDoS: Detection and Response

The bogeyman, the monster under the bed, the creature lurking in the darkness waiting to pounce… 

That’s what it feels like we have made Denial of Service attacks out to be.

What is a Denial of Service attack? 

Pretty self explanatory..   attacks that are directed at removing the ability to use a service.  Most commonly, we see Denial of Service attacks succeed by causing some form of resource exhaustion. This could be exhausting the memory on the application server, filling the network pipes, or a plethora of other options.

How are they done?

Let’s take a walk down memory lane.  Think back, before the age of the Pentium. Imagine the monkeys in front of the obelisk, suddenly, through the quiet you can hear it. The dulcet tones and squeals of the mystical modem reach out for your favorite BBS. It’s a powerhouse BBS, with 5 incoming lines!

Weird…  you can’t connect.   Try again.. and again.. and again.  Still nothing but the busy tone.  Two days pass, you still can’t get on.  Two days without Legend of the Red Dragon? Unthinkable!

Unfortunately, the BBS admin pissed someone off. This someone set their dialer pools to continuously dial and keep all the lines busy.  The BBS has now been effectively Denial of Serviced. Result: One pissed off monkey.

So, returning to modern day, how are DoS attacks done? It depends on what the attacker is attempting to exhaust.  If they aiming for the application, it’s possible to do it with a small (sometimes only 1) attack machine. 

If targeting the network level,  more often than not, attackers will use a "distributed” denial of service. Hence, the dreaded name DDoS. A distributed attack sources from multiple nodes, making it often more difficult to block and detect.

There are many different ways a DoS can occur, so for our purpose we’ll focus today on a simple Network level DoS.

How do we know we are under a Dos Attack?

Some indicators of a potential DoS attack:
Site down
Non-standard connection types (halfopens, etc)
High Connections from Poor IP reputation Sources – There are multiple companies that maintain an IP reputation database.

Indicators to create baselines for: (Baseline being average value over selected period)
Resource usage spike: Memory/CPU on the box averages 40% usage, one tuesday, it suddenly jumps to over 80%
Connection spikes

These are all listed as POTENTIAL indicators, as none of them necessarily mean that you are facing a DoS attack. Any of these hitting would need to be investigated and confirmed.

What do we do if we think we are under attack?

Investigate!

If the site is down, and you see a high amount of tcp-half open connections slamming your connection tables, you have a good probability that you are facing a syn-flood.  

What to look at? 

Connection Tables :

Looking for anomalous connections,  connections not completing, single source or grouped source.

Memory Usage:

Is the memory spiking on the unit?

CPU:

CPU spiked and out of cycles?

Network Captures:

See the truth of what is happening on the wire!

Respond!

Once you have determined you are under a DoS attack, and know the form of the attack, it’s time to implement the response plan. Each attack is slightly different from the last.  The responses have to adapt with the attack.

Let’s do it in lab

I learn better by doing, so, here is a quick DoS lab I was playing with:

The attack du jour is a Syn-Flood.  It’s a well known method and easy to implement as an attacker.

Baselining: Since my network is using a full proxy architecture and the lab is based on a syn-flood attack,  I will do my baseline work on the proxy (LTM). I had some nominal web traffic going across, to simulate users.  I captured connections and cpu usage for this baseline

The Attack: On the Backtrack boxes, I kick up hyenae: hyenae -I 3 -a tcp -f S -s %-%@% -d 00:50:56:a0:16:83-10.4.4.10@80 Which will start sending a flood of syn packets to the virtual.  The effect is pretty instantaneous, cpu spikes and the connection count begins a huge climb.

Detection:

Well, we begin to notice the slight spike in connections, and decide its time to see what is going on here. I could go and and try to view the connection table, but that would add extra strain on the system. The information from the table wouldn’t provide to much value.

To get a gander at what is going on, I snag a tcpdump on the vlan (Ya, I know calling it “attackincoming” is somewhat cheating)

tcpdump -s0 -ni attackincoming -w /var/tmp/capture.pcap
 

The tcpdump shows pretty quick what is going on. This evil attacker is sending in a flood of syn packets and the box is attempting to SYN,ACK back. Of course, for this attack, they don’t have to even complete the handshake. When the SYN of the connection comes in, the server puts the connection into the conn table, and sends it’s SYN,ACK. You can see why sending a flood of SYN’s can effectively overload a box, as each entry takes memory and cpu.

Response:

Welp, we’ve got ourselves a massive SYN flood attack. Time to deal with it.

IP-Source limiting?  Nope, range of IPs

Pull the Plug? Sure, best defense ever, just might anger the investors

Luckily, there is a a known defense for SYN-Flood. Daniel Bernstein came out with a pretty ingenious method of SYN protection. It essentially encodes the connection information in the sequence number of the SYN, ACK, then drops the SYN packet. No mess, no hanging entry in the connection table. (next generation: TCP Cookie Transactions) I’m going to change the SYN checking threshold to 16000. It will allow 16000 connections before implementing SYN Cookies.

Much gooder!