Forum Discussion
Andy_Herrman_22
Nimbostratus
NimbostratusHow to track down rogue persistence values?
My virtual server is using Universal persistence with a somewhat complicated iRule that generates a persistence value from a couple parameters in the URL query string.
In general everything seems to be working. All the connections that require persistence hit the right server. However, I just noticed a bunch of unexpected values being added to the persistence table (seen via `b persist show all`). One of the values seems to be a partial IP address, and I get another one or two that are all non-display characters. I attached a sample output. The only valid persistence value in there is the third ("135680_0").
I added a log message to my iRule every time it sets the persist value for a connection. According to the logs it's never setting those bad values.
Has anyone seen anything like this before? Any hints on how to track down what's going on or what's setting those persistence values?
Sadly I can't post the iRule source right now as I need to get permission before doing so. If there are any specific details that might help with debugging I can probably post those though.
Andy_Herrman_22I found the issue. My iRule wasn't defining persistence values for any connection that didn't need them. It looks like in those cases the F5 would occasionally generate junk persistence values instead of not doing persistence. I added a `persist none` command in the cases that didn't need persistence and the junk values went away.
I'm wondering if I hit a minor bug in the iRule handling code. Based on the persist values I was seeing it looks like something was trying to use an invalid pointer. I'm using LB::detach in my iRule, and I'm wondering if sometimes the F5 is trying to re-use a persistence value that has been deleted, and thus getting whatever data happens to be in memory there at the time.
hoolioCirrostratus
Hi Andy,
Glad you figured out a fix for this. Out of curiosity, what is the actual command you are using to add the UIE persistence record? Can reproduce the issue without the 'persist none' command but log the value of the token before adding it?
Aaron- Andy_Herrman_22Here's a trimmed-down version of my iRule. The `persist none` parts were the only change I made to fix the weird behavior.
when CLIENT_ACCEPTED { set lastPersist "" } when HTTP_REQUEST { Some semi-complicated logic runs to set this, but isn't relevant to the issue. This will either set a real persistence value or one of the constants used by the switch statement later. set persistVal "" Browsers tend to reuse connections when they can. Webservers see each request as separate, but the load balancer works on connections, not requests. Since the requests in the same connection may not share the same perrsistence data we need to do an LB::detach to cause the request to be routed properly. if { $lastPersist != "" and $persistVal != $lastPersist } { LB::detach } set lastPersist $persistVal We only want to set persistence when there is persistence data in the request. Static requests don't need any persistence. We still set the lastPersist value though so that we can detect when the persistence value has changed. switch -- $persistVal { "STATICREQ" { Don't set any persistence for the request persist none } "NOPERSIST" { Don't set any persistence for the request persist none } default { log local0. "HTTP setting persist to: $persistVal" persist uie $persistVal } } }
I think the issue is related to the LB::detach. I'm guessing when the detach happens the string that stores the old persistence value is freed, but if no new one is provided by the iRule (the old behavior) it tries re-using it. Since it has been freed the string is now invalid, and so you get whatever happens to be in memory at that location.