WebLogic information extraction

Our weblogic servers will give me an active list of servers in the X-WebLogic-Cluster-List header if I supply the X-Weblogic-Request-ClusterInfo header with a value of true. This is the missing piece I’ve been trying to identify since weblogic stopped encoding the real server IP in network byte order in version 8. I’d like to populate an array with the servername and the unique arbitrary number it is assigned. By utilizing the rule

when HTTP_REQUEST {
  HTTP::header insert "X-Weblogic-Request-ClusterInfo" "true"
}
when HTTP_RESPONSE {
  if { [HTTP::header exists "X-WebLogic-Cluster-List"] } {
    log "Cluster list: [HTTP::header "X-WebLogic-Cluster-List"]"
    HTTP::header remove "X-WebLogic-Cluster-List"
  }
}

I receive the information below:

-306262974!server_a-93!65535!-1 |

-572170000!server_c-93!65535!-1 |

-851958269!server_b-91!65535!-1 |

-896443913!server_d-93!65535!-1 |

-937189304!server_b-90!65535!-1 |

1685710619!server_c-92!65535!-1 |

1717517013!server_d-90!65535!-1 |

1720885579!server_d-91!65535!-1 |

1949732605!server_b-93!65535!-1 |

296784837!server_c-90!65535!-1 |

324405524!server_c-91!65535!-1 |

472005059!server_d-92!65535!-1 |

534925464!server_b-92!65535!-1 |

548600573!server_a-90!65535!-1 |

82670729!server_a-92!65535!-1 |

894452575!server_a-91!65535!-1

I’m looking for advice on the best way (read: efficient) to extract the servername and the leading arbitrary number associated to it. I know the real IP address for each of those server names, so what I’d like to do is evaluate the JSESSIONID information arriving (jsessionid!primary_server!secondary_server) and send connections to primary_server based on information in the array, and in the LB_FAILED event, send connections to the secondary_server. I’m struggling with getting persistence to behave nicely in failure scenarios and I am looking at alternatives. Any help would be appreciated.

You can use “getfield” to extract the first 3 fields in the header like this:

set jsess [getfield [HTTP::header "X-WebLogic-Cluster-List"] "!" 1]
set server1 [getfield [HTTP::header "X-WebLogic-Cluster-List"] "!" 2]  
set server2 [getfield [HTTP::header "X-WebLogic-Cluster-List"] "!" 3]

Is that what you were looking for…?

HTH

/deb

partially…I would like to first split the content by server:

set server_mappings [split [HTTP::header “X-WebLogic-Cluster-List”] “|” ]

Then set an array with the first two fields in each index value, so that the array looks like this:

{

“-306262974 server_a-93”

“-572170000 server_c-93”

“-851958269 server_b-91”

“-896443913 server_d-93”

“-937189304 server_b-90”

}

However, the ultimate goal is to replace that server_a-93 with a real IP address. I have that information in a class:

class server_name_to_IP {

“server_a-93 10.1.1.10”

“server_c-93 10.1.1.11”

“server_b-91 10.1.1.12”

“server_d-93 10.1.1.13”

“server_b-90 10.1.1.14”

}

So, what I’m trying to get my very tiny brain around is take the header information for each server and build an array so that when I receive requests from clients with cookie or URI with this information:

jsessionid=Gsfsafavlaeruae!-306262974!-851958269

I can extract the 2nd and 3rd values in this information and forward requests appropriately.

I’m not sure if I should just use two arrays, or do a search/replace function, or possible and array of arrays. And of course, the coding skills to get there, of which I have none. So before I spun myself in circles, I thought I’d reach out to the community for some advice.

OK, how about something like this?:

when HTTP_RESPONSE {
  if { [HTTP::header exists "X-WebLogic-Cluster-List"] }{
    set server_mappings [split [HTTP::header "X-WebLogic-Cluster-List"] "|" ]
    init array if non-existent
    array set ::ServerList { }
    clear pre-existing array entries
    (re-initializing as blank array doesn't seem to do it)
    array unset ::ServerList
    foreach server $server_mappings {
      set ::ServerList($server) [getfield [findclass $server $::server_name_to_IP] " " 2]
    }
  }
}

which would create the following ServerList array:

{
-306262974 10.1.1.10
-572170000 10.1.1.11
-851958269 10.1.1.12
-896443913 10.1.1.13
-937189304 10.1.1.14
}

then once you’ve extracted the jsession string (i.e. Gsfsafavlaeruae!-306262974!-851958269) from cookie or URI, you can extract the primary & secondary server IP from the array using the WL server IDs like this:

array get ::ServerList [getfield $jsessionstring "!" 2]
array get ::ServerList [getfield $jsessionstring "!" 3]

HTH

/deb

Still trying to wrap my head around exactly what is transpiring in the array population…I’m a big dummy.

Does getfield start at 0 or 1?

Thanks for the help. I’m doing a rewrite currently and hope to test tomorrow.

Yikes, I see your confusion -- I left out the extraction of the server details needed to build the array. Sorry about that. I think this is more what you are after:

when HTTP_RESPONSE {
  if { [HTTP::header exists "X-WebLogic-Cluster-List"] }{
    set server_mappings [split [HTTP::header "X-WebLogic-Cluster-List"] "|" ]
    init array if non-existent
    array set ::ServerList { }
    clear pre-existing array entries
    (re-initializing as blank array doesn't seem to do it)
    array unset ::ServerList
    foreach servermap $server_mappings {
      set serverID [getfield $servermap "!" 1]
      set serverName [getfield $servermap "!" 2]
      set ::ServerList($serverID) [getfield [findclass $serverName $::server_name_to_IP] " " 2]
    }
  }
}

or to eliminate setting those 2 extra variables, inline substitution gives you:

when HTTP_RESPONSE {
  if { [HTTP::header exists "X-WebLogic-Cluster-List"] }{
    set server_mappings [split [HTTP::header "X-WebLogic-Cluster-List"] "|" ]
    init array if non-existent
    array set ::ServerList { }
    clear pre-existing array entries
    (re-initializing as blank array doesn't seem to do it)
    array unset ::ServerList
    foreach servermap $server_mappings {
      set ::ServerList([getfield $servermap "!" 1]) [getfield [findclass [getfield $servermap "!" 2] $::server_name_to_IP] " " 2]
    }
  }
}

Clear as mud, now, eh?

(getfield starts indexing @ 1, while lindex starts @ 0.)

(edited to correct line-wrapping error)

I finally had some time to work through the rule logic. I think it would have taken me a very long time to figure out that fancy array footwork, thanks for that. I haven’t done the failure or load testing against this yet, but here is the working rule. Basically it makes use of the cluster information provided by Weblogic to map the WebLogic locally significant serverID to an IP address so I can send all connection requests directly to the appropriate primary server without using persistence. It will evaluate the JSESSIONID cookie if present, and the URI for the jsessionid if the cookie is not present.

when CLIENT_ACCEPTED {
  set get_server_mappings 1
}
when HTTP_REQUEST {
   Insert WebLogic Header to retrieve active servers if new client connection
  if { $get_server_mappings } {
     Inserting this header instructs Weblogic to return cluster information
    HTTP::header insert "X-Weblogic-Request-ClusterInfo" "true"
     Reset variable to zero to prevent header insert on every request
    set get_server_mappings 0
  }
   If all servers are inactive, redirect to sorry page on Apache
  if { [active_members myPool] == 0 } {    
    HTTP::redirect "http://[HTTP::header "X-Forwarded-Host"]/unavailable/index.html"
  }
   Replace Host header with X-Forwarded-Host header contents
  if { [HTTP::header exists "X-Forwarded-Host"] } {
    HTTP::header replace "Host" [HTTP::header "X-Forwarded-Host"]
  }
   If JSESSIONID cookie exists, use the contents to extract jsession server reference 
  if { [HTTP::cookie exists "JSESSIONID"] } {
    set serverIP_fromCookie [lindex [ array get ::ServerList [getfield [HTTP::cookie "JSESSIONID"] "!" 2 ] ] 1]
     If an IP was found in the array
    if { $serverIP_fromCookie ne "" } {
       Send the connection to the primary server
      pool myPool member $serverIP_fromCookie
      log "From Cookie: Server $serverIP_fromCookie is primary for jsessionID [getfield [HTTP::cookie "JSESSIONID"] "!" 1 ]"
    } else {
        log "Couldn't extract pool member from array, connection will be load-balanced!"
    }
  } else {
       NO COOKIE, need to extract from URI
       if jsessionid is found in the uri AND in the absense of the jsessionid cookie
      if { [string tolower [HTTP::uri]] contains "jsessionid" } {
         Set IP from list item 1 of the array by extracting jsession server reference  from URI
        set serverIP_fromURI [lindex [ array get ::ServerList [lindex [split [findstr [HTTP::uri] "jsessionid" 11 "?"] "!" ] 1 ] ] 1]
         If an IP was found in the array
        if { $serverIP_fromURI ne "" } {
           Send the connection to the primary server
          pool myPool member $serverIP_fromURI
          log "From URI: Server $serverIP_fromURI is primary for jsessionID [lindex [split [findstr [HTTP::uri] "jsessionid" 11 "?"] "!" ] 0 ]"
        } else {
            log "Couldn't extract pool member from URI, connection will be load-balanced!"
        }
      }
  }
}
when HTTP_RESPONSE {
  if { [HTTP::header exists "X-WebLogic-Cluster-List"] }{
  
     Set indexed variable with server cluster information
    set server_mappings [split [HTTP::header "X-WebLogic-Cluster-List"] "|" ]
     Remove header before returning to client
    HTTP::header remove "X-WebLogic-Cluster-List"
     init array if non-existent
    array set ::ServerList { }
     clear pre-existing array entries by re-initializing as blank array doesn't seem to do it
    array unset ::ServerList
    foreach servermap $server_mappings {
      set ::ServerList([getfield $servermap "!" 1]) [getfield [findclass [getfield $servermap "!" 2] $::intres_servers] " " 2]
    }
  }
}
when LB_FAILED {
  log "LB_FAILED: Server [LB::server addr] failed!"
  set lbfailed 1
   Turn off least connections, set for round robin
  LB::mode rr
   Reselect another pool member. It might be failed server(s) once, but it won’t occur continuously with round robin
  LB::reselect pool myPool
  log "LB_FAILED: New server is [LB::server addr] "
}

No problem, glad I could help.

/deb

I was going to wait until we get through our failure testing as there may be some enhancements on the way…

Well, good news and bad news. Good news is everything works great under normal conditions. Bad news is this logic doesn’t seem to do anything:

when LB_FAILED {
  set failed_server [LB::server addr]
  log local0. "Connection to server $failed_server failed! Reselecting..."
  LB::mode rr
  LB::reselect pool MyPool
  for {set x 0} {$x<4} {incr x} {
    if { [LB::server addr] eq $failed_server } {
      LB::mode rr
      LB::reselect pool MyPool
      log local0. "Attempt $x ==>New server is [LB::server addr]"
    }
  }
}

The log from my testing from a single browser during a failure:

Connection to server 10.1.11.111 failed! Reselecting…

Attempt 0 ==>New server is 10.1.11.111

Attempt 1 ==>New server is 10.1.11.111

Attempt 2 ==>New server is 10.1.11.111

Attempt 3 ==>New server is 10.1.11.111

Also, I see the LB_FAILED event when I am doing single browser testing, but when we do a load test from loadrunner, I don’t see this event trigger. Any ideas why? Any ideas why the server isn’t changing? There are 15 additional resources

From the description in the wiki:

LB::reselect

* Selects the next available member in the current pool, based on pool Load Balancing options

LB::reselect

* Selects the next available member in the specified pool, based on the Load Balancing options of that pool

It appears that the LB::reselect utilizes the pool LB method. How then do I combine the LB::mode command to reselect outside of the normal load balancing method?

Sheesh, this thread is getting long…

Here’s the caveat: The reselect doesn’t actually occur until the LB_FAILED event is finished executing. So, doing an [LB::server addr] in the same execution is going to return the same value, no matter how many times you call LB::reselect. Think of it as indicating that you want to reselect. The LB_FAILED event should get raised again if the new reselected server also fails.

I think I’ll try upgrading first, then open a case if that doesn’t work. I’ll post back results. Thanks for the help.

I upgraded to 9.2.3, and we are now getting the lb_failed event to trigger. All of our 502 errors are now gone, but we still have some weird timeout behavior. Our configuration for testing is:

LoadRunner → F5 webtier → Apache Servers → F5 apptier → WebLogic Servers

The rule is on the F5 apptier, and once loadrunner has ramped up and stabilized in its scenario, we fail a physical server, which kills four of the sixteen weblogic instances. The upgrade eliminated the immediate failures, but loadrunner receives timeouts at the two minute IE timeout. Apache however, waits for, and eventually receives, the 200/302 valid responses from teh F5 apptier. They come in bunches at about 80 sec, 150 sec, 180-190 sec, and 300 sec. Is there anyway to configure around this?

Hi Jason, Can you please tell us what configuration setting you used on Apache / F5 to get it working .We have little different issue but layer are same like your

F5webtier -- Apache --F5AppTier -- Weblogic ( how did you get this working ? ) Please suggest.

Posted By Paresh Birla on 02/09/2011 05:55 AM

Hi Jason, Can you please tell us what configuration setting you used on Apache / F5 to get it working .We have little different issue but layer are same like your

F5webtier -- Apache --F5AppTier -- Weblogic ( how did you get this working ? ) Please suggest.

What’s the issue you’re seeing?

I wasn’t on the apache team, unfortunately. They gave me a set of requirements at functionality they were pulling out and I took it from there. The biggest thing for me was supporting persistence for clients with or without cookies enabled. For those with cookies enabled, we did a simple cookie insert. For those that didn’t, we extracted the jsessionID and stored that and the server IP in the session table. When a request came in, if the inserted cookie was not present, we’d look for the jsession in the URI and then use the host from the session table to map the request accordingly. The logic left in Apache on those apps was whether or not the request was ssl or not since at the time we weren’t offloading SSL on the front-tier LTM.

Hi Jason

This is a really interesting discussion you’ve started here and something I’ve been looking into myself (to simplify the architecture by removing an Apache/OHS layer for load balancing within the application zone).

Did you manage to resolve your issues and get this iRule into production? I’ve been intending to mock up something similar myself so it’s great to see the path has already been trodden but I’m wondering if there’s any sort of fundamental show stopper here.

TIA!

Simon