For more information regarding the security incident at F5, the actions we are taking to address it, and our ongoing efforts to protect our customers, click here.

HTTP slow post mitigation

Problem this snippet solves:

Here is an updated version of the iRule from this post and techtip.

Mitigating Slow HTTP Post DDoS Attacks With iRules – Follow-up https://devcentral.f5.com/Community/GroupDetails/tabid/1082223/asg/50/aft/1174748/showtab/groupforums/Default.aspx#1200083

This past week researchers demonstrated a new HTTP DDoS attack in which a slow POST request will result in leaving a connection open longer than necessary. The heart of the attack relies on sending a POST request with given “content-length” then very slowly sending the POST message body to the server. The server will leave the connection open as it continues to receive data. If a large number of these requests are executed against a server, there is potential for exhausting the connection table leaving the server unable to respond to further requests.

There are a few different ways to mitigate such an attack using iRules. You could build a list of all the URLs that require POST requests and only allow requests for these URLs. This method while effective against a blanket attack would prove ineffective against an attacker who was targeting a page that relies on legitimate POST requests. Aside from that, someone must maintain the list. Enter the “after” command…

The after command was introduced in version 10 of BIG-IP and allows among other things, the cancellation of a previous script. In this example, if our client is sending their POST request too slowly we will instruct our BIG-IP to send them a HTTP 500 error and close the connection after a given period. If their request is received within the allotted time period, we will cancel our previous instruction and allow the connection to them proceed to the origin servers.

Code :

# Based on 'Mitigating Slow HTTP Post DDoS Attacks With iRules' from George Watkins
#
# Requires LTM v10.0+ for the after command

when RULE_INIT {

   # This iRule enforces a minimum length of time ($static::timeout) for a client to send POST request data.
   # The initial values are 2Kb / 2 sec = 1 Kb/s for the first 2Kb.  These values should be tailored for the client base.

   # Default amount of request payload to collect (in bytes).
   # This is the maximum amount of content we will collect.
   # Clients will still be able to send unlimited payload sizes.
   set static::collect_length 2048

   # Default timeout, for POST requests, to send $collect_length bytes (in milliseconds)
   set static::timeout 2000

   # HTML response to send for blocked requests
   set static::block_html {Your POST request is not being received quickly enough. Please retry.}

   # Log debug messages to /var/log/ltm? 1=yes, 0=no.
   set static::post_debug 1
}
when HTTP_REQUEST {

   # Only check POST requests. If the application supports other request methods with payloads, add them in a switch statement here.
   if { [HTTP::method] equals "POST"} {

      if { $static::post_debug } { log local0. "[IP::client_addr]:[TCP::client_port]: POST to [HTTP::host][HTTP::uri]" }

      # Create a local variable copy of the collection amount
      set collect_length $static::collect_length

      # Create a local variable copy of the static timeout
      set timeout $static::timeout

      # Check for a non-existent Content-Length header
      if {[HTTP::header Content-Length] eq ""}{

         # Use default collect length for POSTs without a Content-Length header
         set collect_length $static::collect_length
         if { $static::post_debug } { log local0. "[IP::client_addr]:[TCP::client_port]: No Content-Length value" }

      } elseif {[HTTP::header Content-Length] <= 0}{

         # Don't try to collect a payload if there isn't one
         unset collect_length
         if { $static::post_debug } { log local0. "[IP::client_addr]:[TCP::client_port]: Content-Length: 0." }

      } elseif {[HTTP::header Content-Length] > $static::collect_length}{

         # Use the default collect length
         set collect_length $static::collect_length
         if { $static::post_debug } { log local0. "[IP::client_addr]:[TCP::client_port]: Content-Length: [HTTP::header Content-Length], collecting $collect_length" }

      } else {

         # Collect the actual payload length
         set collect_length [HTTP::header Content-Length]

         # Calculate a custom timeout based on the same ratio we use for the default collect length and default timeout
         set timeout [expr {[HTTP::header Content-Length] * $static::timeout / $static::collect_length }]
         if { $static::post_debug } { log local0. "[IP::client_addr]:[TCP::client_port]: Content-Length: [HTTP::header Content-Length], collecting $collect_length bytes with timeout $timeout ms" }
      }

      # If the POST Content-Length isn't 0, collect (a portion of) the payload
      if {[info exists collect_length]}{

         # If the entire request hasn't been received within X seconds, send a 408, and close the connection
         set id [after $timeout {
            if { $static::post_debug } { log local0. "[IP::client_addr]:[TCP::client_port]: $timeout ms reached. Closing connection" }
            HTTP::respond 408 content $static::block_html Connection Close
            TCP::close 
         }]

         # Trigger collection of the request payload
         HTTP::collect $collect_length
         if { $static::post_debug } { log local0. "[IP::client_addr]:[TCP::client_port]: Collecting $collect_length" }
      }
   }
}
when HTTP_REQUEST_DATA {

   if { $static::post_debug } { log local0. "[IP::client_addr]:[TCP::client_port]: Collected [HTTP::payload length] bytes." }

   # Check if the 'after' ID exists
   if {[info exists id]} {

      # If all the POST data has been received, cancel the connection closure
      if { $static::post_debug } { log local0. "[IP::client_addr]:[TCP::client_port]: Canceling \$id: $id" }
      after cancel $id
      # This needs to be reworked to address ID454692
      # unset id
   }
}
Published Mar 18, 2015
Version 1.0

1 Comment

  • Are there any concerns about race conditions for Keep-Alive connections? It seems the 'id' variable is scoped to the TCP connection. If we get request A and begin data collection (and therefore initiated the 'after' call with id 1) and before the collection is finished, we receive request B and collection is started and another 'after' call is started with id 2. When HTTP_REQUEST_DATA is called for the collection of request A, wouldn't $id now be 2 and therefore the cancellation call for the after event 1 would not happen. Could this cause termination of valid connections in some edge cases? Edit: I over thought it. HTTP 1.1 doesn't allow concurrent outstanding requests over a single connection, so no race condition.