DDoS: Detection and Response

The bogeyman, the monster under the bed, the creature lurking in the darkness waiting to pounce… 

That’s what it feels like we have made Denial of Service attacks out to be.

What is a Denial of Service attack? 

Pretty self explanatory..   attacks that are directed at removing the ability to use a service.  Most commonly, we see Denial of Service attacks succeed by causing some form of resource exhaustion. This could be exhausting the memory on the application server, filling the network pipes, or a plethora of other options.

How are they done?

Let’s take a walk down memory lane.  Think back, before the age of the Pentium. Imagine the monkeys in front of the obelisk, suddenly, through the quiet you can hear it. The dulcet tones and squeals of the mystical modem reach out for your favorite BBS. It’s a powerhouse BBS, with 5 incoming lines!

Weird…  you can’t connect.   Try again.. and again.. and again.  Still nothing but the busy tone.  Two days pass, you still can’t get on.  Two days without Legend of the Red Dragon? Unthinkable!

Unfortunately, the BBS admin pissed someone off. This someone set their dialer pools to continuously dial and keep all the lines busy.  The BBS has now been effectively Denial of Serviced. Result: One pissed off monkey.

So, returning to modern day, how are DoS attacks done? It depends on what the attacker is attempting to exhaust.  If they aiming for the application, it’s possible to do it with a small (sometimes only 1) attack machine. 

If targeting the network level,  more often than not, attackers will use a "distributed” denial of service. Hence, the dreaded name DDoS. A distributed attack sources from multiple nodes, making it often more difficult to block and detect.

There are many different ways a DoS can occur, so for our purpose we’ll focus today on a simple Network level DoS.

How do we know we are under a Dos Attack?

Some indicators of a potential DoS attack:
Site down
Non-standard connection types (halfopens, etc)
High Connections from Poor IP reputation Sources – There are multiple companies that maintain an IP reputation database.

Indicators to create baselines for: (Baseline being average value over selected period)
Resource usage spike: Memory/CPU on the box averages 40% usage, one tuesday, it suddenly jumps to over 80%
Connection spikes

These are all listed as POTENTIAL indicators, as none of them necessarily mean that you are facing a DoS attack. Any of these hitting would need to be investigated and confirmed.

What do we do if we think we are under attack?

Investigate!

If the site is down, and you see a high amount of tcp-half open connections slamming your connection tables, you have a good probability that you are facing a syn-flood.  

What to look at? 

Connection Tables :

Looking for anomalous connections,  connections not completing, single source or grouped source.

Memory Usage:

Is the memory spiking on the unit?

CPU:

CPU spiked and out of cycles?

Network Captures:

See the truth of what is happening on the wire!

Respond!

Once you have determined you are under a DoS attack, and know the form of the attack, it’s time to implement the response plan. Each attack is slightly different from the last.  The responses have to adapt with the attack.

 

Let’s do it in lab

I learn better by doing, so, here is a quick DoS lab I was playing with:

The attack du jour is a Syn-Flood.  It’s a well known method and easy to implement as an attacker.

Syn-Flood:

What it be:  A flood of syn packets sent to a server

Why it works:  The server puts an entry in the connection table for the new connection and sends out the Syn,Ack. At a point, no new connections can be made, as the resources on the server are exhausted.

The Lab:

2 Backtrack Systems

Virtual LTM v11.1  4 core, 8gb Ram

Apache Application Server

The Setup

Easy network:

10.4.4.0/24- public network

10.4.4.5 – Backtrack 1

10.4.4.6 – Backtrack 2

10.4.4.10 – Basic Virtual

10.5.5.0/24- internal network

10.5.5.10 – Pool Member

Just for testing purposes, I raised the syn-check protection to 6.5 million.

 

Baselining:

Since my network is using a full proxy architecture and the lab is based on a syn-flood attack,  I will do my baseline work on the proxy (LTM). I had some nominal web traffic going across, to simulate users. 

I captured connections and cpu usage for this baseline

The Attack:

On the Backtrack boxes, I kick up hyenae:

hyenae -I 3 -a tcp -f S -s %-%@% -d 00:50:56:a0:16:83-10.4.4.10@80

Which will start sending a flood of syn packets to the virtual. 

The effect is pretty instantaneous, cpu spikes and the connection count begins a huge climb.

 

Detection:

Well, we begin to notice the slight spike in connections, and decide its time to see what is going on here. I could go and and try to view the connection table, but that would add extra strain on the system. The information from the table wouldn’t provide to much value.

To get a gander at what is going on, I snag a tcpdump on the vlan (Ya, I know calling it “attackincoming” is somewhat cheating)

tcpdump -s0 -ni attackincoming -w /var/tmp/capture.pcap
 

 

The tcpdump shows pretty quick what is going on. This evil attacker is sending in a flood of syn packets and the box is attempting to SYN,ACK back. Of course, for this attack, they don’t have to even complete the handshake. When the SYN of the connection comes in, the server puts the connection into the conn table, and sends it’s SYN,ACK. You can see why sending a flood of SYN’s can effectively overload a box, as each entry takes memory and cpu.

Response:

Welp, we’ve got ourselves a massive SYN flood attack. Time to deal with it.

IP-Source limiting?  Nope, range of IPs

Pull the Plug? Sure, best defense ever, just might anger the investors

 

Luckily, there is a a known defense for SYN-Flood. Daniel Bernstein came out with a pretty ingenious method of SYN protection. It essentially encodes the connection information in the sequence number of the SYN, ACK, then drops the SYN packet. No mess, no hanging entry in the connection table. (next generation: TCP Cookie Transactions) I’m going to change the SYN checking threshold to 16000. It will allow 16000 connections before implementing SYN Cookies.

Much gooder! 

 

 

 
Published May 15, 2012
Version 1.0

Was this article helpful?

4 Comments

  • Joon_Ho_7885's avatar
    Joon_Ho_7885
    Historic F5 Account
    Good contents,

     

     

    I'm Joon at F5 Korea, the issue here is that the volume of TCP SYN flooding attack (ACK, FIN, FIN/ACK flooding, etc.) is 10Gbps level.

     

    In case of HTTP GET Flooding, the volume is about 1~3Gbps.

     

     

    Would you please share what the volume of TCP SYN flooding attack during your test? i'm just curious to compare the CPU % of before vs. after using SYN checking threshold.

     

     

    Thanks!
  • Joon:

     

     

    Take a look at the image next to "The Attack". At that point, the system was seeing about 1.7 Million ( 1,700,000) connections.

     

     

    You can set Hyenae to send a specific count of attempts with a :

     

     

    -c [Min packet count]

     

    -C [Max packet count]

     

     

    Keep in mind, this test was a standard syn flood. If I am reading right, it sounds like you are actually seeing fully established connections (complete 3way handshake) and then a GET request.

     

     

    If that is so, the syn "cookie" protection would not be in play, since the handshake is being completed.

     

     

    In the case of a full handshake HTTP GET DoS we'd have to look at the traffic and investigate what identifiable characteristics it has.

     

     

    Feel free to ping me if I can help!

     

     

    -Josh
  • Joon_Ho_7885's avatar
    Joon_Ho_7885
    Historic F5 Account
    @Josh,

     

     

    I meant two kinds of attacks.

     

    One is TCP SYN Flooding attack and another is HTTP GET Flooding attack separately.

     

     

    Yes, you are right. About HTTP GET Flooding attack, TCP SYN cookie is not a workaround because HTTP GET packet is sent to server after TCP-3W-Handshaking process.

     

     

     

  • However, I think, you may want to set Virtual Server conn. limits as stated here:

     

     

    http://support.f5.com/kb/en-us/solutions/public/7000/300/sol7301.html?sr=23494670

     

     

    regards,

     

    HH