Unexpected TMOS utalsiation hit

I have a bigip 6400 platform running LTM v 9.4.7

 

 

We have an iRule to direct HTTP requests coming into the F5 from the datacentre out to the appropriate external service (For XML messaging). This is a lot cleaner than having 250 vips for 250 different services.

 

The rule matches the HTTP host to a pool defined on the server, so if we need to update a host, we can do this via the web interface, without making any changes to the iRule by hand (= lower risk, easier operation).

 

Slap the iRule on the virtual server and a serverssl profile on there and you've got encrypted traffic going out.

 

 

We've also got some error handling to deal with people trying to get to services they shouldn't from this particular vip, and error messages if the external service is down.

 

 

Here is the code:

 

 iRule to direct HTTP requests based on hostname    
   when HTTP_REQUEST {   
   Extract hostname - needs to be lower case to match pool name   
   set host [string tolower [HTTP::host]]   
    Check if the hostname (and therefore the pool name the request will be sent to   
    end with correct domain - Prevent using this VS to hop out a different pool   
   if { ($host ends_with ".bob.com") and !($host contains ".test.")} {   
   if { [catch { pool $host } ] } {   
    no matching pool name - so move on to error   
   HTTP::respond 404 content "Endpoint not defined"     
   }   
   } else {   
      
   HTTP::respond 403 content "Invalid URL"     
   }   
      
   }   
      
   when LB_FAILED {   
   HTTP::respond 504 content "Endpoint Unavailable"     
   }

 

 

So far, so good. We've tested, we've run iRule benchmarking and its reasonably efficient...

 

 

config  cat /proc/cpuinfo   
   model name      : AMD Opteron(tm) Processor 246   
   stepping        : 1   
   cpu MHz         : 1992.276   
      
   config  bigpipe rule Prod_Spine_Generic_Routing show all   
   RULE Prod_Spine_Generic_Routing   
   +-> HTTP_REQUEST   3123 total   0 fail   0 abort   
   |   |     Cycles (min, avg, max) = (17437, 51642, 92343)   
   +-> LB_FAILED   0 total   0 fail   0 abort   
       |     Cycles (min, avg, max) = (0, 0, 0)

 

 

When that's spreadsheeted it comes back with 40,000 TPS, which is perfectly reasonable for the amount of traffic we're expecting.

 

 

So at that point we said, we've got this sorted, put it into production and carried on to the next piece of work.

 

 

We've now got people using this VIP, and during their first bulk load run they got to the dizzying heights of 12 HTTP requests per second. During this period the TMOS utilisation moved from its usual 1-2% to 25%, and when they backed off to 6 TPS it reduced to 10-12%.

 

 

Clearly somethings not right here, but the timing stats (Produced from production data) show its all fine. There has to be some sort of hidden cost somewhere, but I can't see any obvious place its coming from.

 

Our reseller isn't being very helpful, so I'm between a rock and a hard place here.

 

 

Any thoughts would be most gratefully received...