Problem with persistence

I have a setup with a BIG-IP load balancer with 2 servers in a pool being load balanced. The URI that the user is connecting with contains a query parameter that is used for the persistence value. According to our logging the iRule is correctly extracting the value and setting the persistence, but requests with the same value don’t always go to the same machine.

The iRule is pretty simple:

when HTTP_REQUEST {
  set fulluri [HTTP::uri]
  set uid [findstr [HTTP::uri] "uid=" 5 "&"]
  if { $uid != "" } {
    log local0. "Set persist value to uid: $uid  | $fulluri"
    persist uie $uid
  }
  else {
    log local0. "No uid found in HTTP REQUEST  | $fulluri"
  }
}

The logging that displays always shows the correct uid value. However, I get logging on both servers showing that they’ve gotten requests, and those logs show the same uid. BIG-IP stats also show that it’s sending requests to both servers, thought the persistence table seems correct.

Has anyone run into anything like this before? I can’t find anything wrong with the iRule, and the requests are happening fast enough that the persistence values wouldn’t be timing out.

That could explain it.

We can have multiple clients who can connect with the same uid value, and they should all go to the same server. The code you have there looks like it’ll do the add for each individual client, instead of just once for that specific value. Will the add call have any problems if it’s already in the table, or will it just ignore it? Is there a way to only do that add if the uid isn’t already there instead of once per client (again, I may be misinterpreting things)

Hmm…not sure if it’s possible (or how) to compare values that are already in the table…

As long as it’s adding the same record I wouldn’t see a problem with that, but that’s probably not as efficient as one would hope.

Denny

As long as it doesn’t reset the persistence (change the server it points to) then it should be fine.

Looks like the persist function has a get method. I could probably use that to query to see if the uid is already in the table and if so just skip it, but I’m not sure what the format of the returned stuff is (how to tell if there’s no value or not). I’m guessing just checking if it is equal to “” would be enough. Does anyone know?

It doesn’t seem to like the

persist add uie $uid

call. I’m getting the following error in the log:

TCL error: Rule UID_Persist - Prerequisite operation not in progress (line 1) invoked from within “persist add uie $uid”

Can anyone help me out here? I have no idea what that error means.

Here’s the iRule code I currently have:

when CLIENT_ACCEPTED {
  set add_persist 1
}
when HTTP_REQUEST {
  set fulluri [HTTP::uri]
  set uid [findstr [HTTP::uri] "uid=" 5 "&"]
  if { $uid != "" } {
    if { $add_persist } {
      log local0. "Adding persist value to table: $uid | $fulluri"
      persist add uie $uid
      set add_persist 0
    }
    else {
      log local0. "Set persist value to uid: $uid  | $fulluri"
      persist uie $uid
    }
  }
  else {
    log local0. "No uid found in HTTP REQUEST  | $fulluri"
  }
}

I’ve only seen the persist add statement on the serverside events.

From my experience, the general idea seems to be:

1) there must be a response event in which the persistence value is extracted from the response to create the persistence record;

and

2) there must be a request event in which the persistence value submitted in each request is compared to the persistence table

3) a rule containing that logic must be applied within a persistence profile, rather than applied directly to the virutal as a resource.

The error you’re getting /may/ be caused by applying the rule directly to the virtual. Try applying the rule to a persistence profile instead, then apply the persistence profile to the virtual.

The persistence profile will manage the timeout of the record, so from what I understand you need to re-add the persistence record on each response to prevent the record from timing out on an active session.

Here’s an example of the rule structure and the placement of the 2 “persist” commands: Click here

The same persistence record may be shared by multiple clients submitting a persistence token with the same value. Persistence record looks something like “persist_value pool_member”, so any request coming in with that persistence value will use that node.

HTH

/deb

The persistence value isn’t generated in a response. The persistence value is part of all connections to the server, including the very first one.

Also, you’re saying the timeout for the persistence value is a timeout on when it was added? I thought it was a timeout on when it was last accessed, so just doing the

persist uie $uid

should restart it.

Not sure about the applying to virtual vs persistence profile. I believe it’s applied to the profile, but I’m not sure (I’m not the one doing the configuration, just the one writing the iRules).

I changed the iRule to do the add in the response. According to the logs it’s adding the correct value, but it’s happening every time someone connects (which might be ok, I’m not sure). Unfortunately, the persistence still isn’t working. The connections with the same uid are going to different servers.

I did verify that the iRule is applied to the persistence profile, so that part at least is correct.

Any ideas why it wouldn’t be persisting? Is there anything else I might be missing?

Here’s the current iRule I’m using:

when HTTP_REQUEST {
  set fulluri [HTTP::uri]
  set uid [findstr [HTTP::uri] "uid=" 5 "&"]
  if { $uid != "" } {
    log local0. "Set persist value to uid: $uid  | $fulluri"
    persist uie $uid
  }
  else {
    log local0. "No uid found in HTTP REQUEST  | $fulluri"
  }
}
when HTTP_RESPONSE {
  if { $uid != "" } {
    log local0. "Adding persist value to table: $uid"
    persist add uie $uid
  }
}

Try using the CLIENT_ACCEPTED event that I used earlier to limit the addition of a record to the first connection.

Also because you aren’t setting $uid in the HTTP_RESPONSE event, I don’t think anything’s getting added, and you’re seeing the log from the HTTP_REQUEST event.

I don’t think order matters, but it might make it easier to see the logic if you move the response event before the request event, since it’s on the first response that you want to set the persist record.

when CLIENT_ACCEPTED {
  set add_persist 1
when HTTP_RESPONSE {
   set uid [findstr [HTTP::uri] "uid=" 5 "&"]
   if { $uid != "" and $add_persist} {
   log local0. "Adding persist value to table: $uid"
   persist add uie $uid
   set add_persist 0
}
when HTTP_REQUEST {
   set fulluri [HTTP::uri]
   set uid [findstr [HTTP::uri] "uid=" 5 "&"]
   if { $uid != "" } {
   log local0. "Using persist value uid: $uid | $fulluri"
   persist uie $uid
  } else {
     log local0. "No uid found in HTTP REQUEST | $fulluri"
  }
}

This still may not help with the fact that you said you have different clients using the same uid however.

Denny

The value for $uid was still in scope during the HTTP_RESPONSE message, so it had the correct value (same reason the add_persist value keeps it’s value of 1 between CLIENT_ACCEPTED and HTTP_REQUEST). The logging messages I put in place verified this.

Also, the CLIENT_ACCEPTED stuff wouldn’t really work for my case as we can have multiple clients using the same $uid (the uid indicates an application instance, and that application can be shared between users).

Actually, I’m going to take a few steps back, going back to the original iRule. When using this iRule we *are* seeing the persistence table records created correctly, so it looks like just doing the `persist uie $uid` call is adding it to the table correctly. While monitoring it the persistence values themselves are correct, but BIG-IP doesn’t seem to be honoring it. The persistence is set up, but it’s doesn’t seem to be affecting the connections.

Ok, this is *really* weird.

I modified the iRule to log what the current node in the persistence table is just before it does the `persist uie $uid` call. I also ssh’ed into the machine so I could get dumps of the persistence table. I then ran a few test connections with various uid values and between each I looked at the logs and the persistence table. I waited at least 10 seconds between each request so I could tell if the uid persistence timeout was being reset (indicating that that persistence was used). I generated a file that shows the logs for the request (there are 2 lines per request), then the results of `bigpipe persist show all` on the machine. I added the results of this below (edited a bit so the lines are so long). Here’s the rule I used:

when HTTP_REQUEST {
  set fulluri [HTTP::uri]
  set uid [findstr [HTTP::uri] "uid=" 4 "&"]
  if { $uid != "" } {
    log local0. "Set persist value to uid: $uid  | $fulluri"
    set currentPersist [persist lookup uie $uid node]
    log local0. "Current persist for $uid is $currentPersist"
    persist uie $uid
  }
  else {
    log local0. "No uid found in HTTP REQUEST  | $fulluri"
  }
}

What I found was that the first request and UID worked fine. The first request showed that there wasn’t a value for the uid, and subsequent requests showed that there was. This is correct.

Weird things started happening when I changed the uid value. All the later requests showed logging that it was finding and calling `persist uie $uid` with the correct values. However, the persistence record wasn’t being added to the list, and the timeouts on the *old* uid was was being reset, suggesting it was using that persistence record instead. This continued to happen, even if I changed the uid value again.

The only time it correctly used the new uid value seemed to be when I waited long enough for the old one to timeout and be removed (this is what happened with the last test).

Does anyone have any ideas what could be causing this? At this point it looks like a problem with how BIG-IP is handling the call to `persist uie $uid`, as if it’s ignoring the value I’m giving it.

PERSISTENT CONNECTIONS --
No persistent connections were found.
Jun 22 15:31:34 : Set persist value to uid: 1  | /testapp?uid=1&sc=0
Jun 22 15:31:34 : Current persist for 1 is
PERSISTENT CONNECTIONS --
    Mode: universal   Value: 1
    Virtual: 10.10.70.171:http    Node: 10.10.70.32:http    Age:  2sec
Jun 22 15:31:41 : Set persist value to uid: 1  | /testapp?uid=1&sc=0
Jun 22 15:31:41 : Current persist for 1 is 10.10.70.32
PERSISTENT CONNECTIONS --
    Mode: universal   Value: 1
    Virtual: 10.10.70.171:http    Node: 10.10.70.32:http    Age:  5sec
Jun 22 15:31:50 : Set persist value to uid: 1  | /testapp?uid=1&sc=0
Jun 22 15:31:50 : Current persist for 1 is 10.10.70.32
PERSISTENT CONNECTIONS --
    Mode: universal   Value: 1
    Virtual: 10.10.70.171:http    Node: 10.10.70.32:http    Age:  1sec
Jun 22 15:32:03 : Set persist value to uid: 432  | /testapp?uid=432&sc=0
Jun 22 15:32:03 : Current persist for 432 is
PERSISTENT CONNECTIONS --
    Mode: universal   Value: 1
    Virtual: 10.10.70.171:http    Node: 10.10.70.32:http    Age:  15sec
Jun 22 15:32:10 : Set persist value to uid: 432  | /testapp?uid=432&sc=0
Jun 22 15:32:10 : Current persist for 432 is
PERSISTENT CONNECTIONS --
    Mode: universal   Value: 1
    Virtual: 10.10.70.171:http    Node: 10.10.70.32:http    Age:  9sec
Jun 22 15:32:17 : Set persist value to uid: 432  | /testapp?uid=432&sc=0
Jun 22 15:32:17 : Current persist for 432 is
PERSISTENT CONNECTIONS --
    Mode: universal   Value: 1
    Virtual: 10.10.70.171:http    Node: 10.10.70.32:http    Age:  2sec
Jun 22 15:32:28 : Set persist value to uid: 321  | /testapp?uid=321&sc=0
Jun 22 15:32:28 : Current persist for 321 is
PERSISTENT CONNECTIONS --
    Mode: universal   Value: 1
    Virtual: 10.10.70.171:http    Node: 10.10.70.32:http    Age:  14sec
Jun 22 15:32:34 : Set persist value to uid: 321  | /testapp?uid=321&sc=0
Jun 22 15:32:34 : Current persist for 321 is
PERSISTENT CONNECTIONS --
    Mode: universal   Value: 1
    Virtual: 10.10.70.171:http    Node: 10.10.70.32:http    Age:  7sec
Jun 22 15:35:38 : Set persist value to uid: 90  | /testapp?uid=90&sc=0
Jun 22 15:35:38 : Current persist for 90 is
PERSISTENT CONNECTIONS --
    Mode: universal   Value: 90
    Virtual: 10.10.70.171:http    Node: 10.10.70.39:http    Age:  3sec

Sheesh, I go on vacation one day and no one can seem to answer this for you…

The problem is that the persist command does not change the already connected back-end server on a subsequent request over the same connection. In order for that to happen, you could either enable OneConnect (add the oneconnect profile to your virtual) or modify the rule to call LB::detach if the uid value changes. Then the subsequent request would utilize the new persist setting to (re)connect to the appropriate server.

Also, just to be clear about the “persist add” command. In this case it is certainly not needed. That is because the “persist uie” command will add a record if it does not exist. Normally, the “persist add” command is only necessary if the first server response contains the information necessary to persist on (EG, as is the case with JSESSION type persistence).

Anyway, hope this helps and it’s not too late.

Oops, minor bug.

In the case where no UID was given but it wasn’t a static page (which is really an error, but needs to be handled gracefully) add:

set persistVal "NOUID"

I am unclear why you would sent the persist value to NOID or STATICREQ and then persist using those values. If there is no UID presented, why would you create or use a persistence record for that request?