Forum Discussion

frank_combopia1's avatar
frank_combopia1
Icon for Nimbostratus rankNimbostratus
Feb 15, 2011

iRule + profiles driving up resources

I have an active/standby pair of stock, 1500s (v9.4.6) front-ending two Internet proxy services. I'm using a single VS, letting an iRule decide which service pool to direct the request. Persistence is a requirement of one the services, and I have source address affinity assigned.

 

 

Prior to introducing the iRule and the second service, the VS was configured simply with a TCP profile, and although the number of connections are high, around 80k, resource utilization was very low.

 

 

 

The iRule requires http inspection, so I enabled the http and OneConnect Host-based profiles on the VS.

 

 

 

The number of connections hasn't changed, still around 80k, but since enabling the iRule with the additional profiles, resource utilization has gone through the roof. TMM quickly rises to it's 90% cap and stays there all day. We're investigating some failover events and RST messages from the ltm logs, and performance is perceived to be degraded.

 

 

 

My question is: Can I modify any of the profile attributes, or the iRule to alleviate the high resource consumption?

 

 

 

Thanks!

 

 

 

The iRule:

 

 

 

when RULE_INIT {

 

Debug toggle- zero = off, one = on

 

set ::DEBUG 0

 

Bluecoat pool HTTP-PROXY_80

 

PITC pool HTTP-PROXY-IRONPORT

 

}

 

when HTTP_REQUEST {

 

if { [matchclass [IP::client_addr] equals ::Pilot_Subnets] } {

 

if {$::DEBUG} {log local0.info "MozyX-1.0 PITC Client [IP::client_addr]"}

 

if { [HTTP::header "User-Agent"] contains "mozycorp-windows/1.8" } {

 

if {$::DEBUG} {log local0.info "MozyX-1.1 Caught Old Mozy: [HTTP::header "User-Agent"]"}

 

if {$::DEBUG} {log local0.info "MozyX-1.2 Method [HTTP::method]"}

 

if {$::DEBUG} {log local0.info "MozyX-1.3 Host: [HTTP::host]"}

 

if {$::DEBUG} {log local0.info "MozyX-1.4 URI: [HTTP::uri]"}

 

pool HTTP-PROXY_80 member 3.56.142.241 80

 

} else {

 

if {$::DEBUG} {log local0.info "MozyX-2.0 [HTTP::uri]"}

 

pool HTTP-PROXY-IRONPORT

 

}

 

} else {

 

if {$::DEBUG} {log local0.info "MozyX-3.0 Not a PITC client [IP::client_addr] sent [HTTP::host]"}

 

pool HTTP-PROXY_80

 

}

 

}

 

when LB_SELECTED {

 

if {$::DEBUG} {log local0.info "MozyX-4.0 LB_Selected Chose [LB::server pool], [LB::server addr]"}

 

}

 

 

application delivery
dev
devops
iRules
  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Feb 15, 2011
    If it's such a big performance hit, you could create a separate virtual server for the pilot subnets and not use an iRule on either virtual server. Else, you could optimize the iRule slightly by removing the debugging altogether: 

    
when HTTP_REQUEST {
   if { [matchclass [IP::client_addr] equals Pilot_Subnets] } {
      if { [HTTP::header "User-Agent"] contains "mozycorp-windows/1.8" } {
         pool HTTP-PROXY_80 member 3.56.142.241 80
      } else {
         pool HTTP-PROXY-IRONPORT
      }
   } else {
      pool HTTP-PROXY_80 
   }
}

    Aaron
  • frank_combopia1's avatar
    frank_combopia1
    Icon for Nimbostratus rankNimbostratus
    Feb 15, 2011
    Thanks for the quick reply. I neglected to mention I'm using one VS so I can reuse the existing PAC file installed on these 1000s of clients.

     

     

    I see now there's a lot of conditionals applied with the debugging, even if disabled. I'll drop them immediately.

     

     

     

    Thanks!

     

     

     

  • Chris_Miller's avatar
    Chris_Miller
    Icon for Altostratus rankAltostratus
    Feb 16, 2011
    Frank - did you have a chance to reconfigure the rule? I'm curious what kind of CPU reduction you saw.

     

  • frank_combopia1's avatar
    frank_combopia1
    Icon for Nimbostratus rankNimbostratus
    Feb 16, 2011
    Thanks for following up. I did; I stripped to the bare minimum. Unfortunately, there's no difference in TMM utilization, it's pegged at 90% all day.

     

     

    Tonight, I'm going to remove the OneConnect Host profile and see if that buys us anything. We've been investigating some ltm log messages that keep leading us to Solution articles that refer to OneConnect as a possible culprit.
  • Chris_Miller's avatar
    Chris_Miller
    Icon for Altostratus rankAltostratus
    Feb 16, 2011
    Posted By frank combopiano on 02/16/2011 02:02 PM

     

    Thanks for following up. I did; I stripped to the bare minimum. Unfortunately, there's no difference in TMM utilization, it's pegged at 90% all day.

     

     

    Tonight, I'm going to remove the OneConnect Host profile and see if that buys us anything. We've been investigating some ltm log messages that keep leading us to Solution articles that refer to OneConnect as a possible culprit.

     

    What do the LTM logs look like?
  • frank_combopia1's avatar
    frank_combopia1
    Icon for Nimbostratus rankNimbostratus
    Feb 17, 2011
    The ltm log has about 140 of these over the last 16 hours:

     

     

    Feb 16 17:49:00 tmm tmm[1693]: 011f0007:3: http_process_state_prepend - Invalid action EV_INGRESS_DATA during ST_HTTP_PREPEND_HEADERS (Server side: vip=VS_3.56.240.250_HTTP profile=http pool=HTTP-PROXY_80)

     

     

    and a few of these that are making us nervous:

     

     

    Feb 16 11:01:33 GETYGMUSHADCX01-PROXY sod[1712]: 010c0025:5: Toggle from active to standby to active.

     

     

    Another thing we're struggling with is connection mirroring. It's enabled in the persistence profile associated with the VS mentioned in the message. We're hesitant to remove it because most if not all of the clients using one of these two proxy services would be required to re-authenticate after a fail-over. At least that's what we believe; it's based on unique digital certs at each proxy.

     

     

  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Feb 17, 2011
    Hi Frank,

    I didn't think you'd gain too much by removing the debug checks from the iRule. It was more a clean up suggestion.

    The active/standby/active log event is definitely a cause for concern. This is possibly due to an issue with the load and/or configuration.

    Do you have connection mirroring enabled (on a virtual server or SNAT)? Or do you have persistence mirroring enabled? Both will add load, but I think connection mirroring will add a lot more. If you have connection mirroring enabled, that would be the first thing I'd consider disabling (before OneConnect or other changes). Connection mirroring doesn't add much value for the relatively short lived TCP connections that HTTP uses.

    If this is HTTP proxy traffic you could also consider using CARP which is stateless and requires no mirroring:

    SOL11362: Overview of the CARP hash algorithm

    http://support.f5.com/kb/en-us/solutions/public/11000/300/sol11362.html

    Also, I'd suggest opening a case with F5 Support so you can get a detailed analysis done of the logs and configuration. This is most likely impacting production traffic, so troubleshooting via the forums might not be fast enough.

    Lastly, you can optimize the iRule a little more by only performing the IP check once per TCP connection. Again, I don't think this will make a big difference, but it is more efficient. 

    
when CLIENT_ACCEPTED {

    Only check once per TCP connection if the client IP is part of the datagroup
   if { [matchclass [IP::client_addr] equals Pilot_Subnets] } {
      set pilot 1
   } else {
       Assign the standard pool now
      set pilot 0
      pool HTTP-PROXY_80 
   }
}
when HTTP_REQUEST {

    Check if the client IP 
   if {$pilot == 1 }{
      if { [HTTP::header "User-Agent"] contains "mozycorp-windows/1.8" } {
         pool HTTP-PROXY_80 member 3.56.142.241 80
      } else {
         pool HTTP-PROXY-IRONPORT
      }
   }
}

    Aaron
  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Feb 17, 2011
    Or you could squeeze out a few more cycles by disabling HTTP_REQUEST for matching IP addresses (assuming you don't have any other iRules running on the virtual server that use the HTTP_REQUEST event):

     

     

    
when CLIENT_ACCEPTED {

    Only check once per TCP connection if the client IP is part of the datagroup
   if { not [matchclass [IP::client_addr] equals Pilot_Subnets] } {

       Assign the standard pool now and prevent HTTP_REQUEST code from running
      pool HTTP-PROXY_80 
      event HTTP_REQUEST disable
   }
}
when HTTP_REQUEST {

    This code only gets executed if the client IP was in the datagroup
   if { [HTTP::header "User-Agent"] contains "mozycorp-windows/1.8" } {
      pool HTTP-PROXY_80 member 3.56.142.241 80
   } else {
      pool HTTP-PROXY-IRONPORT
   }
}

     

     

    Aaron