For more information regarding the security incident at F5, the actions we are taking to address it, and our ongoing efforts to protect our customers, click here.

Forum Discussion

AlanTen's avatar
AlanTen
Icon for Altostratus rankAltostratus
Sep 04, 2015

One-Way Persistence records problem

Hello All,

tl;dr - I have an iRule that is allowing HTTP_Request to insert invalid persistence records and unnecessary updates to existing persistence records and I would like to stop that by using 'persist lookup uie' instead of 'persist uie', but I think I'm doing it wrong 😞 Jump to the iRules to win a free headache, or not 😄

Admittedly I'm still getting to grips with some of the more funky things you can do with iRules and my current challenge is that a customer of our rolled out a new client app version which has caused the VS that supports the app to chew up around 46% of the system resources where it used to use around 8%.

Thankfully everything is functioning just fine however if someone so much as farts the system could run out of resources and failover which would be a massive problem. Aside from the fact that we know the client side app has changed behavior in some way the symptoms we see on the F5, aside from the CPU load, is HUGE amounts of persistence record updates across the HA Sync interface, in the region of 5Gb/20 seconds.

What I am attempting to achieve is three fold, one, to reduce the number of updates to the persistence table, two, ditch the invalid JSESSIONIDs that are being inserted and finally, try save some cpu time.

The current universal persistence iRule that is currently in use allows, to my understanding at least, the HTTP Request to update or add persistence records by using the usual 'persist uie' command:

 iRule Name: JSESSIONID_persist
when HTTP_REQUEST {
    Check if there is a JSESSIONID cookie
   if { [HTTP::cookie "JSESSIONID"] ne "" }{
       Persist off of the cookie value with a timeout of 5 minutes (300 seconds)
      persist uie [string tolower [HTTP::cookie "JSESSIONID"]] 300
   }
}
when HTTP_RESPONSE {
    Check if there is a jsessionid cookie in the response
   if { [HTTP::cookie "JSESSIONID"] ne "" }{
       Persist off of the cookie value with a timeout of 5 minutes (300 seconds)
      persist add uie [string tolower [HTTP::cookie "JSESSIONID"]] 300
   }
}

The design doc for the solution says that only the app servers can supply a valid JSessionID and so this action by the user is not necessary and has two problems, 1. the HTTP_Requests are able to insert or update stale JSessionIDs in the persistence table and 2. the overhead caused by the lookup and add combo that 'persist uie' carries out as well as the addition and tracking of these stale records is completely unnecessary.

So my intention is to only allow lookups to be run by the HTTP_Requests which I was hoping would result in a successful direction of the connection to the correct pool member that had created the matching persistence record:

 iRule Name: JSESSIONID_persist_v2
when HTTP_REQUEST {
 Check if there is a JSESSIONID cookie in the request header.
if { [HTTP::cookie "JSESSIONID"] ne "" }{
 Run a lookup for any existing records, return relevant values for any matches 
 otherwise ignore and follow load balancing method to select destination server.
persist lookup uie [string tolower [HTTP::cookie "JSESSIONID"]]
}
}
when HTTP_RESPONSE {
 Check if there is a jsessionid cookie in the response.
if { [HTTP::cookie "JSESSIONID"] ne "" }{
 Persist off of the cookie value with a timeout of 5m15s to ensure that 
 the F5's persistence entry will not expire before the server token.
persist add uie [string tolower [HTTP::cookie "JSESSIONID"]] 315
}
}

This worked great in load and functions tests that we ran in the dev environment where the VS had two pools, but failed on the production test VS where there are 7 pool members and it looks like the persistence is not being followed.

I have a sneaky suspicion that my basic understanding of the 'persist lookup' function is wrong and that it does not make load balancing decisions for the connection based on the returned info which would mean that it was complete chance that the load and function tests worked perfectly.

If this is the case then I think I need to assign the returned node info to a variable and then use that to direct the connection to the pool member using 'node $returned_lookup' or some such. Is this right or is there a better way to do this?

The main thing is that this iRule has to be low cost because it has to handle a large volume of operations, I was even looking to see if I could use 'switch' to take over from the 'if' statement, but I got a bit stuck with the negative comparison.

What do you guys and gals reckon?

Regards, Al

8 Replies

  • giltjr's avatar
    giltjr
    Icon for Nimbostratus rankNimbostratus

    persist lookup does not make load balancing decisions, it just looks to see if there is a persist record that matches what you are using for a key. In fact the none of the persist options make any load balancing decisions, they just lookup, add, delete a "record" of persistence.

     

    I'm fairly sure that switch is no more efficient than a single if.

     

    One possible issue is that you set the lifetime of the persistence record to 315 seconds. So after 315 seconds of no activity from the user, the persistence record will be deleted and the next time they come in the request will be re-load balanced.

     

    Is it possible that there are instances where there is more than 315 seconds between requests from the user?

     

    Is it possible that there are instance where it could take longer than 315 seconds for a response?

     

    On responses servers normally only add cookies when the cookie does not appear on the request. Meaning that the cookie on the response will only appear once, the first time the server sees a request. If a response takes longer than 315 seconds, then the F5 will delete the persistence record, the response will NOT have the JESSIONID cookie, and so the persist command will not be executed.

     

  • giltjr's avatar
    giltjr
    Icon for Nimbostratus rankNimbostratus

    Had to do some more research because I started to remember some "weirdness" when I wrote my first iRule using persist uie dealing with timeout.

     

    When you do a persist add with a timeout the persist record will be deleted after that number of seconds unless the timeout is "refreshed."

     

    The timeout will be refreshed only if you do another "persist add" or just a plain "persists". A "persist lookup" will not refresh the timeout. Since the cookie will NOT exist on any response other than the 1st one, you have to refresh the persist record in HTTP_REQUEST. This is why the sample iRule use just plain "persist", this will look up to see if a record exists and if it does it resets the timeout to what you have coded.

     

    So in your "new" iRule, 315 seconds after the 1st request comes in the persist record will be deleted, because you never refresh the timeout. You need to have just plain "persist" in HTTP_REQUEST, if the cookie JSESSIONID exists.

     

    However my other notes about the time out still applies. If there is more than 315 seconds between users requests, then the persist record will be deleted and then next request will be re-load balanced.

     

  • giltjr's avatar
    giltjr
    Icon for Nimbostratus rankNimbostratus

    I guess or posts crossed. If the server kills the JESSIONID after 300 seconds, then the original iRule was messing things up because it was refreshing the persist record on every request that came in with 300 seconds of the last request.

     

    Doing what you changed to last might be the only way to do this.

     

  • Are you by any chance running Oracle Weblogic as your backend servers?

     

    If so you can reduce the session table to 7 records flat.

     

  • The iRule that we successfully tested on Friday had some minor adjustments made because somewhere in the back end something was wanting the pool name as well 😕

     iRule Name: JSESSIONID_Persist_v4
     Last Mod: AT 2015/09/04
    
    when HTTP_REQUEST {
     Check if there is a JSESSIONID cookie in the request header.
    if { [HTTP::cookie "JSESSIONID"] ne "" }{
     Run a lookup for any existing records, return relevant values for any matches 
     otherwise ignore and follow load balancing method to select destination server.
    set ppool [ persist lookup uie [string tolower [HTTP::cookie "JSESSIONID"]] pool ]
    set pnode [ persist lookup uie [string tolower [HTTP::cookie "JSESSIONID"]] node ]
    set pport [ persist lookup uie [string tolower [HTTP::cookie "JSESSIONID"]] port ]
    if { $pnode != "" }{
    pool $ppool member $pnode $pport
    }
    }
    }
    when HTTP_RESPONSE {
     Check if there is a jsessionid cookie in the response.
    if { [HTTP::cookie "JSESSIONID"] ne "" }{
     Persist off of the cookie value with a timeout of 6 minutes to ensure that 
     the F5's persistence entry will not expire before the server token.
    persist add uie [string tolower [HTTP::cookie "JSESSIONID"]] 315
    }
    }
    
  • Here is a cut down version of our persistence iRule.

    It offers a more robust handling of persistence:

    * stale session cookie

    * no session cookie

    * allows debug logs (on persistence or request level)
     Weblogic Persist iRule, Version 0.93
     Mars, 2012
     Last update: Aug, 2012
     Updated v11.6.0: 15/02/2015
    
     Created by Opher Shachar (contact me through devcentral.f5.com)
    
     Purpose:
     This iRule persists requests based on the JSESSIONID cookie issued by the
     Weblogic server.
    
    when RULE_INIT {
        Set debug level to: 0, 1 or 2
       set static::Weblogic_debug 1
        Set a timeout of 5.25 minutes (315 seconds)
       set static::Weblogic_timeout    315
    }
    
    when CLIENT_ACCEPTED {
       set log_prefix "Client [IP::client_addr]:[TCP::client_port]"
    }
    
    when HTTP_REQUEST {
        Log details for the request
       if {$static::Weblogic_debug > 1} {
          log local0. "$log_prefix: Request to [HTTP::uri] with cookie: [HTTP::cookie value $static::Weblogic_CookieName]"
       }
    
        Check if there is a session cookie
       set pkey [HTTP::cookie $static::Weblogic_CookieName]
    
        Did we find a session id?
       if {$pkey ne ""} {
           Persist off of the session id value with the set timeout
          if {[persist lookup uie $pkey] ne ""} {
             persist uie $pkey $static::Weblogic_timeout
             if {$static::Weblogic_debug > 1} {
                log local0. "$log_prefix: Used persistence record from cookie or path: [persist lookup uie $pkey]"
             }
          } else {
              Client gave stale session id  --  don't persist
             persist none
             if {$static::Weblogic_debug > 0} {
                log local0. "$log_prefix: No persistence record found for key: $pkey"
             }
          }
       } else {
           Client gave no session id  --  don't persist
          persist none
          if {$static::Weblogic_debug > 1} {
             log local0. "$log_prefix: No session id given in cookie or path."
          }
       }
    }
    
    when HTTP_RESPONSE {
        Check if there is a session cookie in the response
       set pkey [HTTP::cookie $static::Weblogic_CookieName]
    
        Did we find a session id?
       if {$pkey ne ""} {
           Persist the session id value with the set timeout
          persist add uie $pkey $static::Weblogic_timeout
          if {$static::Weblogic_debug > 0} {
             set val [persist lookup uie $pkey]
             log local0. "$log_prefix: Added persistence record from cookie or redirect: $pkey -> $val"
          }
       }
    }
    

    Cut out are:

    * persistence on JVMID alone (reduce persistence table size)

    * support for cookie-less user-agents

    * workaround for bug in pre-v11.1.0 HF2
  • giltjr's avatar
    giltjr
    Icon for Nimbostratus rankNimbostratus

    One thing you may want to look into and test with to make sure it does not upset your application server is to set the expires options on the JSESSIOINID cookie, or better yet see if the application server can set it.

     

    That way the only time a inbound request will have the cookie is when the cookie is still valid in the application server. It would mean a bit more logic in your iRule.

     

    In HTTP_RESPONSE you would do something like:

     

    if { [HTTP::cookie "JSESSIONID"] ne "" }{
             Set the cookie to expire 290 seconds from now.
            HTTP::cookie expires JSESSIONID 290 relative
             Persist off of the cookie value with a timeout of 4m50s to ensure that 
             the F5 will never send an expired cookie back to the application.
            persist add uie [HTTP::cookie "JSESSIONID"] 290
        }

    You might have to experiment, I'm not 100% sure that the above example will add the expired information, you might have to save the value of the current JSESSSIONID cookie, delete the existing one and then insert the cookie.

     

    I also took the "string tolower" out, this removes the overhead of converting the JSESSIONID to all lower case letters. So you will need to take it out of the HTTP_REQUEST section of code. It will also remove the possibility of having two JSESSIONID's that are different due to case (JSESSIOND=AbCd and JSESSIOND=aBcD) end up being used in a persistence record as the same because they are both converted to "abcd".

     

    I used 290 seconds instead of 315. I personally think it is better for the browser and the F5 to timeout the cookie sooner than the back-end application. That way a request will never get back to the application with an expired JSESSIONID.

     

    Again, if you can get the application server to set the expires date, that would be best then you only have to worry about the persistence record, but still set it just slightly less than the applications time.

     

  • giltjr's avatar
    giltjr
    Icon for Nimbostratus rankNimbostratus

    Quick question, is there a special reason you are using uie for persistence? Why not just setup a persistence profile that uses a cookie, set it to a non-session cookie have it's lifetime be 240 seconds, or 315, or whatever you choose.