Forum Discussion

bchick2_8645's avatar
bchick2_8645
Icon for Nimbostratus rankNimbostratus
Nov 21, 2011

Limit Client Connections with Table

We have been using (verbatim) the iRule given on the Wiki at Click Here to successfully limit the number of concurrent connections from a single source IP address. However, we recently learned the use of global variables in the rule was demoting our LTM 1600 out of CMP mode. Some research seemed to indicate that replacing the global array with the new (at least new to me) table command was the right approach. I took a first shot at that and it seemed to be working fine at first but recently we started receiving log messages as follows:

Unable to resume pending rule event CLIENT_CLOSED on closed flow XXX.XXX.XXX.XXX:XXXX->YYY.YYY.YYY.YYY:443

Where the X's are the client source IP address / port and the Y's are the server IP. This wasn't happening at first but started over time. Didn't notice it before the upgrade from 10.2.2 HF3 to 10.2.3 but won't swear that it didn't start until then. Searching devcentral and the support site I can't find any hits on that particular log entry. Can anybody tell me what I might be doing wrong? I'm pasting the text of the iRule below, any help confirming the validity of the iRule or pointers for improving it would be greatly appreciated.

Thanks in advance.


when RULE_INIT {
     The maximum number of TCP connections to the virtual server from a single client IP address
    set static::max_connections_per_ip 1000
     Table name (will later be appended with VS name)
    set static::tbl "vsratelimit"
}
when CLIENT_ACCEPTED {
    
     If client IP is exempted skip the rest of the rule and return
    if {[class match [IP::client_addr] equals concurrent_sessions_exemptions]} {
        return
    }
    
     Set the key to the client IP
    set key "[IP::client_addr]"
     Append the VS name to the end of the table name
    set tbl ${static::tbl}_[virtual name]
    
     If the IP is already in the table increment the count
    if {[table keys -subtable $tbl] contains $key} {
        set value [table lookup -subtable $tbl $key]
        
         Check if over the max connection limit and if so reject
        if {$value >= $static::max_connections_per_ip} {
            log local0. "Max connections exceeded for IP: $key, rejecting connection"
            reject
        } else {
        
             Increment the connection count
            table incr -subtable $tbl $key
        
             Only for debug:
            log local0. "Count for $key incremented to $value"
        }
    } else {
         IP is not already in the table so add it with count 1
        table set -subtable $tbl $key 1 indef
        log local0. "Key added to table for $key"
    }
}
when CLIENT_CLOSED {
    
     If client IP is exempted skip the rest of the rule and return
    if {[class match [IP::client_addr] equals concurrent_sessions_exemptions]} {
        return
    }
    
     Set the key to the client IP
    set key "[IP::client_addr]"
     Append the VS name to the end of the table name
    set tbl ${static::tbl}_[virtual name]
  
     Check if the client has a count in the table
    if {[table keys -subtable $tbl] contains $key} {
         Decrement the count by 1
        table incr -subtable $tbl $key -1
         Check if the count is 0 or negative
        set value [table lookup -subtable $tbl $key]
        log local0. "Count for $key decremented to $value"
        if {$value <= 0} {
             Clear the table entry
            table delete -subtable $tbl $key
            log local0. "Table entry deleted for $key"
        }
    }
}

  • are you able to get packet trace while log is happening?
  • I'll work on getting that. The problems are easiest to replicate when we're under heavy load from simulated load testing which makes catching just the relevant traffic kind of tricky since there's so much. I'll see what I can do.
  • I thought you'd always have the option of doing clean up in the CLIENT_CLOSED event. But this error seems to indicate that sometimes the context for the iRule or table access from it is gone when the event fires. If you don't get a reply from a developer on this shortly, I'd try opening a case to get clarification on why it's happening. If you do open a case, can you post the case number so we can check in on it?

     

     

    For support: the logging of the error (as opposed to the actual condition occurring) might have been added in 10.2.3 as noted in BZ364130.

     

     

    Thanks, Aaron
  • I opened case C998162 on this. Will still try to troubleshoot a little on my own as time permits but we have a lot going on today so I just went ahead

     

    and submitted it. Will let you know what comes of it.

     

     

    Thanks
  • spark_86682's avatar
    spark_86682
    Historic F5 Account
    There have been some issues with suspending commands (like the table command) and connections that are in the process of being removed. That error message was added to 10.2.3 to note when that problem was hit (instead of silently failing like before). You should continue your support case to get a resolution for your issue, as it should eventually get to a developer with more information than I have, and it would be great if you could post that here.

     

     

    That said, I think that your iRule could use some improvement. It uses multiple suspending commands when one would often suffice (which may be contributing to your problem), and it decrements shared counters. See the examples at http://devcentral.f5.com/Default.aspx?tabid=63&articleType=ArticleView&articleId=2381, specifically the ones under "Events that are happening right now".
  • Hamish's avatar
    Hamish
    Icon for Cirrocumulus rankCirrocumulus
    Ah... yeah.. Lightbulb moment... Head rush... Room is spinning... That strikes a chord... IIRC with 20.20 hindsight I had a similar case open last year with checkpoint support when I was working on the LDAP performance measurement iRule and software. Sorry,. I'm at a different client now, so I don't have access to my email train with the support call.

     

     

    But basically the result of it was that CLIENT_CLOSED isn't guaranteed to fire... So relying on a decrement in the event doesn't work.

     

     

    H
  • Okay, I guess that makes sense. If that's the case then the iRule from the Wiki entry that I linked in my original post would have the same problem since it also relies on the CLIENT_CLOSED event to decrement the count. So based on the example in the link posted by spark it looks like I would have to modify my iRule to have a separate table for each client IP address and use the client port as the key and then look at how many keys are in the table to determine the number of clients from that IP. Then rather than decrementing a count I just remove the key when the CLIENT_CLOSED event fires or otherwise rely on the timer to clear it.

     

     

    I still need to read through that a little more to make sure I'm not missing anything but is that the basic idea that you're suggesting?
  • Here is my rewrite based on the approach I mentioned above. I have not had a chance to test it yet so I apologize if there is something obvious wrong with it. I will test it later today but based on the examples in the link posted by spark I think this is a better approach. After I have a chance to run it I will let you know whether the log entries go away and whether it appears to otherwise work okay.

    Please let me know if you see any issues with this approach. I really appreciate the help.

    when RULE_INIT {
    
         The maximum number of TCP connections to the virtual server from a single client IP address
        set static::max_connections_per_ip 1000
         Table name (will later be appended with VS name)
        set static::tbl "vsratelimit"
    
    }
    
    when CLIENT_ACCEPTED {
    
     If client IP is exempted skip the rest of the rule and return
    if {[class match [IP::client_addr] equals concurrent_sessions_exemptions]} {
    return
    }
    
     Set the key to the client TCP port
    set key "[TCP::client_port]"
     Append the VS name and client IP to the end of the table name
    set tbl ${static::tbl}_[virtual name]_[IP::client_addr]
    
     Check if over the max connection limit and if so reject
    if {[table keys -subtable $tbl -count] >= $static::max_connections_per_ip} {
    
     Reject the connection
    event CLIENT_CLOSED disable
    reject
    log local0. "Connection rejected for [IP::client_addr], max connections exceeded"
    } else {
    
     Connection accepted, add key for client port
    table set -subtable $tbl $key 1 180
    log local0. "Entry added to $tbl for $key"
     Set timer to keep key alive, if connection dies without CLIENT_CLOSED the
     timer will die and the timeout on the key will cause it to be removed
    set timer [after 60000 -periodic { table lookup -subtable $tbl $key }]
    }
    }
    
    when CLIENT_CLOSED {
    
     If client IP is exempted skip the rest of the rule and return
    if {[class match [IP::client_addr] equals concurrent_sessions_exemptions]} {
    return
    }
    
     Set the key to the client IP
    set key "[TCP::client_port]"
     Append the VS name to the end of the table name
    set tbl ${static::tbl}_[virtual name]_[IP::client_addr]
        
     Remove the timer
    after cancel $timer
     Clear the table entry
    table delete -subtable $tbl $key
    log local0. "Table $tbl entry deleted for $key"
    } 
  • spark_86682's avatar
    spark_86682
    Historic F5 Account
    That is a much better approach. Nice work! Couple improvements could be made, mostly because we haven't updated the example iRule in that article (sorry). First, the key and tbl local variables should carry over from CLIENT_ACCEPTED to CLIENT_CLOSED, so there's no need to compute them again there. Second, to help protect against the issue you originally encountered, I'd delete the table entry in CLIENT_CLOSED before canceling the timer. Finally, if you have a critical need to never exceed the limit even by a little, then you should do the table set before getting the key count and then delete the entry you just added if you are over limit and the reject the connection.
  • Thanks, I've made those changes and have started running with it on a dev box. At first glance it appears to be working fine so I'll just let it run and see how it behaves over time.