Universal Persistence with X-forwarder

Hi Sumanta,

to [persist uie] your clients by either using an existing X-Forwarded-For header (prefered method) or by using the [IP::client_addr] of the underlying TCP session (fallback method), you may take a look to the rather simple iRule below...

when HTTP_REQUEST { 
    if { [set orig_ip [HTTP::header value "X-Forwarded-For"]] eq "" } { 
        set orig_ip [getfield [IP::client_addr] "%" 1]
    } 
     Tweak the persistence table timeout as needed. 1800 is the configured duration in seconds.
    persist uie $orig_ip 1800
}

Note: Issue a tmsh show /ltm persistence persist-records mode universal to list the currently active uie persistence records.

Cheers, Kai

Hi Yann

Can you please assist with the error debug I pasted? Sorry to bother you so much.

01070151:3: Rule [/Common/iRule_rate_limit] error: Unable to find value_list (URI_LIST_TO_LIMIT) referenced at line 7: [class match [string tolower [HTTP::uri]] ends_with URI_LIST_TO_LIMIT]

Regards,

Sumanta.

Hi Yann

What about this one? Is it same as your code logic or different?

From http://devcentral.f5.com/wiki/iRules.table.ashx

Limit each client IP address to 20K concurrent connections

when CLIENT_ACCEPTED {

 Max connections per client IP

set limit 20000 

 Set a subtable name with a standard prefix and the client IP

set tbl "connlimit:[IP::client_addr]"

 Use a key of the client IP:port

set key "[IP::client_addr][TCP::client_port]"

 Check if the subtable has over X entries

if { [table keys -subtable $tbl -count] >= $limit } {

  log local0. "[IP::client_addr]:[TCP::client_port]: Rejecting connection ([table keys 

  -subtable $tbl -count] connections / limit: $limit)"

  reject

} 

else {


    Add the client IP:port to the client IP-specific subtable 

      with a max lifetime of 1260 seconds (30min), matched with L4 profile

   table set -subtable $tbl $key "ignored" 1260

   log local0. "[IP::client_addr]:[TCP::client_port]: Allowing connection ([table keys 

    -subtable $tbl -count] connections / limit: $limit)"

}

}

 when CLIENT_CLOSED {

 When the client connection is closed, remove the table entry

 table delete -subtable $tbl $key

 log local0. "[IP::client_addr]:[TCP::client_port]: Decrementing ([table keys -subtable $tbl 
 -count] connections / limit: $limit)"


}

Hi Yann

Now the URI i rule with XFF is giving error while loading into the F5. Is there any syntax validation system online so that we can verify the rules before pasting?

Aug 17 11:50:12 VSESITE-99-LB01 err mcpd[8325]: 01070151:3: Rule [/Common/iRule_rate_limit-uie] error: /Common/iRule_rate_limit-uie:10: error: [missing a script after "if"][ ] Aug 17 11:50:37 VSESITE-99-LB01 err mcpd[8325]: 01070151:3: Rule [/Common/iRule_rate_limit-uie] error: /Common/iRule_rate_limit-uie:10: error: [missing a script after "if"][ ] Aug 17 11:51:51 VSESITE-99-LB01 err mcpd[8325]: 0107167d:3: Data publisher not found or not implemented when processing request (unknown request), tag (20594). Aug 17 11:52:51 VSESITE-99-LB01 err mcpd[8325]: 01070151:3: Rule [/Common/iRule_rate_limit-uie] error: /Common/iRule_rate_limit-uie:5: error: [parse error: missing close-brace][{ if { [HTTP::header exists X-forwarded-for] } { set client_IP_addr [getfield [lindex [HTTP::header values X-Forwarded-For] 0] "," 1] } set client_IP_addr [IP::client_addr] if { ([HTTP::method] eq "GET") and ([class match [string tolower [HTTP::uri]] ends_with URI_LIST_TO_LIMIT] ) } { set getcount [table lookup -notouch "$client_IP_addr:[HTTP::uri]"] if { $getcount equals "" } { table set "$client_IP_addr:[HTTP::uri]" "1" $static::timeout $static::timeout } else { if { $getcount < $static::maxReqs } { table incr -notouch "$client_IP_addr:[HTTP::uri]" } else { reject } } }] /Common/iRule_rate_limit-uie:6: error: [command is not valid in the current scope][if { [HTTP::header exists X-forwarded-for] } {

Hi Yann

I finally got the URI related iRule working as well. Only thing left is the testing for both virtual servers against DOS attacks. Since I am using real IP via XFF, so I lowered the maxReqs to 200.

ltm rule iRule_rate_limit-uie {

when RULE_INIT {
set static::maxReqs 200;
set static::timeout 3600;
}
 when HTTP_REQUEST {
   if { [HTTP::header exists X-Forwarded-For] } {
       set client_IP_addr [getfield [lindex  [HTTP::header values X-Forwarded-For]  0] "," 1]
    } else {
        set client_IP_addr [IP::client_addr]
    }
    if { ([HTTP::method] eq "GET") and ([class match [string tolower [HTTP::uri]] ends_with URI_LIST_TO_LIMIT] ) } {
             if { [class match [IP::client_addr] equals ips_whitelist] }{
                return
             }
        set getcount [table lookup -notouch "$client_IP_addr:[HTTP::uri]"]
        if { $getcount equals "" } {
            table set "$client_IP_addr:[HTTP::uri]" "1" $static::timeout $static::timeout
        } else {
            if { $getcount < $static::maxReqs } {
                table incr -notouch "$client_IP_addr:[HTTP::uri]"
            } else {
                reject
            }
        }
      }
    }
  }

Hi Sumanta,

you may try one of the iRules below. They will either track the HTTP request or TCP connections per timeframe by just using a single table call. The expected RAM consumption should be very little...

iRule to track the TCP connection / timeframe

when RULE_INIT {
     set static::maxReqs 20000;
     set static::timeout 1800;
}
when CLIENT_ACCEPTED {
    if { [set count [table incr -mustexist "DoS_[IP::client_addr]" "1"]] ne "" } then {
        if { $count < $static::maxReqs } then {
             Allow the request
        } elseif { $count == $static::maxReqs } then {
            log -noname local0.debug "DoS Protection: Client \"[IP::client_addr]\" has reached its HTTP request limits. Blocking the client for the next \"[table lifetime -remaining "DoS_[IP::client_addr]"]\" seconds."
            reject
        } else {
            reject
        }
    } else {
        table set "DoS_[IP::client_addr]" "1" indefinite $static::timeout
    }
}

iRule to track the HTTP request / timeframe

when RULE_INIT {
     set static::maxReqs 20000;
     set static::timeout 1800;
}
when HTTP_REQUEST {
    if { [set count [table incr -mustexist "DoS_[IP::client_addr]" "1"]] ne "" } then {
        if { $count < $static::maxReqs } then {
             Allow the request
        } elseif { $count == $static::maxReqs } then {
            log -noname local0.debug "DoS Protection: Client \"[IP::client_addr]\" has reached its HTTP request limits. Blocking the client for the next \"[table lifetime -remaining "DoS_[IP::client_addr]"]\" seconds."
            reject
        } else {
            reject
        }
    } else {
        table set "DoS_[IP::client_addr]" "1" indefinite $static::timeout
    }
}

Cheers, Kai

Hi Kai

Thanks for the updated codes. I got some reference from DC and used the below code. Till now it appears to work fine and meet my requirements.

when CLIENT_ACCEPTED {
  set tbl "connlimit:[IP::client_addr]"
  set key "[TCP::client_port]"
  table set -subtable $tbl $key "ignored" 180
  if { [table keys -subtable $tbl -count] > 30000 } {
    table delete -subtable $tbl $key
    event CLIENT_CLOSED disable
    reject
  } else {
    set timer [after 1260 -periodic { table lookup -subtable $tbl $key }]
  }
}
when CLIENT_CLOSED {
  after cancel $timer
  table delete -subtable $tbl $key
}
}

Let me see how far it lasts the DOS attack tests. Probably, I'll get to test your code later on.

Regards,

Sumanta.