My iRule Impressions & Optimiziation-Request-"Contest" (IP to STOREID)

[sre_81614]

Hello community, first of all I have to say two 4 words: I love our BIGIPs! 🙂 But when it comes to "iRule-Scripting", I see some disadvantages not regarding tcl itself, but the implementation of tcl in LTM which brings some limitations with it. (documented here: https://support.f5.com/kb/en-us/solutions/public/6000/300/sol6319.html ). For Example, this disallows "iRulers" like me from using user defined procedures (using the proc command). This is the biggest disadvantage compared to "native" tcl scripting, because you have to write down the same function every time you need it in your script. I know there are already discussions (for example at http://devcentral.f5.com/Default.aspx?tabid=53&forumid=5&tpage=1&view=topic&postid=2156428332 ) and also Customer Requests (CR96170) asking F5 to allow the proc command! I've also contacted our F5 representive refering to this request. Not documented - but also unusable/non-functional - are some math commands like "pow"! ^^ Said that, let me talk about my real matter of concern, and why I opened this thread. I'm working at a retail company in Germany, which got 1400 stores/branchoffices connected to our headquarter. These stores are using webapplications (let's call it the store-intranet) hosted at our headquarter. All this stores, got a unique IP address, which can be calculated on a static algorithm. In the past, the webapplication server knew this algorithm to identfiy the store id by the client ip address or X-Forwarded-For Header including the client ip address. That means that all applications (some written by our own developers, some written by 3rd parties) always had to know the algorithm to calculate the store id based on the ip to enforce permissions/views for the specific store requesting a page/element! Now, having our lovely BIGIPs, we moved this "ip2storeid"-functionality into the loadbalancer (using iRules!), so modifications on this algorithm only have to be done on one central place. This saves a lot of time and prevent implementation errors on the application(server)side. Simply said, the loadbalancer takes the ip, calculates the storeid based on the given algorithm and inserts a X-Store-Id in to the request header before sending it to the application servers. Works like a charme! BUT! (there's always a but :-))... After evualating the iRule perfomance, it shows up that this algorithm takes "a few" more cpu cycles! This part of the iRule takes an 450000cycles/request average!! So I'd like to ask all you "iRulers" out there to take a little challenge on my script, to save cpu cycles ! (but obtain all functionality and errorhandling as already implemented!)... Because of.. ME == networkenginer-by-heart && ME != programmer-by-heart I guess there is a lot of potential, to save cpu cycles in this part of my I rule. 'Nuff said, here's the part of my iRule I've talked (too much!?) about: Set X-VKST-ID set ip [IP::client_addr] Setting the ip for testing purposes because dev workstation does not belong to store-network 😉 set ip 10.192.18.0 if {[scan $ip "%d.%d.%d.%d" a b c d] == 4} { foreach i "$a $b $c $d" { if {($i > 255) || ($i < 0)} { return "" } } set iplong [expr {$a} << 24 | {$b} << 16 | {$c} << 8 | {$d}] if {$iplong < 0} { set iplong [expr 4294967296 + {$iplong}] } } set net_start 172.20.0.0 if {[scan $net_start "%d.%d.%d.%d" a b c d] == 4} { foreach i "$a $b $c $d" { if {($i > 255) || ($i < 0)} { return "" } } set net_startlong [expr {$a} << 24 | {$b} << 16 | {$c} << 8 | {$d}] if {$net_startlong < 0} { set net_startlong [expr 4294967296 + {$net_startlong}] } } log local0. "net_startlong1 is $net_startlong" set net_size 262144 set net_segment_size 32 if { $iplong >= $net_startlong && $iplong < ( $net_startlong + $net_size ) } then { log local0. "netversion 1 found." set VKSTID [expr ( {$iplong} - {$net_startlong} ) / {$net_segment_size} ] if {[HTTP::header exists X-Store-id]} { HTTP::header replace X-Store-Id $VKSTID } else { HTTP::header insert X-Store-Id $VKSTID } } set net_start 10.192.0.0 if {[scan $net_start "%d.%d.%d.%d" a b c d] == 4} { foreach i "$a $b $c $d" { if {($i > 255) || ($i < 0)} { return "" } } set net_startlong [expr {$a} << 24 | {$b} << 16 | {$c} << 8 | {$d}] if {$net_startlong < 0} { set net_startlong [expr 4294967296 + {$net_startlong}] } } log local0. "net_startlong2 is $net_startlong" set net_size 4194304 set net_segment_size 512 if { $iplong >= $net_startlong && $iplong < ( $net_startlong + $net_size ) } then { log local0. "netversion 2 found." set VKSTID [expr ( {$iplong} - {$net_startlong} ) / {$net_segment_size} ] if {[HTTP::header exists X-Store-id]} { HTTP::header replace X-Store-Id $VKSTID } else { HTTP::header insert X-Store-Id $VKSTID } } The intention for you to help me on this is easy! Save cpu cycles = save power consumption = save planet earth! Long live the green IT! 🙂

[Nat_Thirasuttakorn]

Hi saschareuter, I think the follow modifications may help.... - use IP::addr command to manipulate IP address and validate if the address is valid (you may see example here:- http://devcentral.f5.com/wiki/default.aspx/iRules/IP__addr.html) Here is an example iRule that I believe equivalent to yours.... when RULE_INIT { Setting the ip for testing purposes because dev workstation does not belong to store-network 😉 set ip 10.192.18.0 set net_start 172.20.0.0 set net_maskbit 14 set net_mask 0.3.255.224 0.3.255.224 = 00000000-00000011-11111111-11100000 net_size = 262144 so ignore first 14 bits net_segment_size = 32 so ignore last 5 bits set net_segment_bit 5 if { [IP::addr $ip/$net_maskbit equals $net_start] }{ log local0. "net $net_start found." scan [IP::addr $ip mask $net_mask] "%d.%d.%d.%d" a b c d set VKSTID [expr ( {$a} << 24 | {$b} << 16 | {$c} << 8 | {$d} ) >> {$net_segment_bit} ] log local0. "VKSTID = $VKSTID" } set net_start 10.192.0.0 set net_maskbit 10 set net_mask 0.63.254.0 set net_segment_bit 9 if { [IP::addr $ip/$net_maskbit equals $net_start] }{ log local0. "net $net_start found." scan [IP::addr $ip mask $net_mask] "%d.%d.%d.%d" a b c d set VKSTID [expr ( {$a} << 24 | {$b} << 16 | {$c} << 8 | {$d} ) >> {$net_segment_bit} ] log local0. "VKSTID = $VKSTID" } } - combine some command together - pre-calculate some value in RULE_INIT event and store it in global variable for reuse (for example, value related to 172.20.0.0 and 10.192.0.0) - current version of BIG-IP does not support function (proc) but I believe you might be able to reduce number of lines of irule by using foreach and array variable. Nat

[sre_81614]

Hey Nat, first off all, thanks for the reply 🙂 I've tried the modified iRule you provided. Functionality is still given (!), and it looks more beautiful. That's all great, but the impact on the cpucycles/perfomance is still very high (i would say somethin' like my first shot)... can you confirm that? I'll try to combine some line, but for now it looks it still uses the same cpu cycles for processing... 😞 Greets, Sascha

[Nat_Thirasuttakorn]

Hi Sascha, in my basic test, I have seen cpu cycle reduced. what I did was putting partial of your iRule and mine in RULE_INIT (of separate iRule) and compare cpucycle reported. If you still see that cpucycle is very high. Maybe I was missing something. can you post your tested iRule? May I ask what hardware/software are you using? (I tested on 6800/v10HF1 and 3600/v9.4.6) btw, in production irule, you may remove unnecessary log (or consolidate them by saving all information in variable first then issue log command at the end). I guess you may already did this. (so just in case) Nat

[sre_81614]

Hey Nat, here is the whole iRule I use including the "ip to storeid" function. It takes an average of 400.000cpu cycles/request. Putting some of the IP convert functions into the RULE_INIT doesn't work on my BIGIP1600 running V10 HFA2, because [IP::client_addr] isn't known while init. I'm looking forward to see if you can get this rule to burn less cpucycles 😉 Thanks in advance! - SR

 
  
  
 timing on 
  
 when RULE_INIT { 
 } 
  
 when HTTP_REQUEST { 
  
      Set default publication and pool 
 set publication vkstintranet 
 pool vkstintranet 
  
  Set publications 
  Syntax: publicpath internalpath pool 
  
     set publications { 
 publicpath1internalpath1appsrv1-8020 
 publicpath2internalpath2appsrv2-8020 
 publicpath3internalpath3appsrv3-8020 
 publicpath4internalpath4appsrv4-8020 
 publicpath5internalpath5appsrv5-8020 
 publicpath6internalpath6appsrv6-8020 
 publicpath7internalpath7appsrv7-8020 
 publicpath8internalpath8appsrv8-8020 
 publicpath9internalpath9appsrv9-8020 
 publicpath10internalpath10appsrv10-8020 
 publicpath11internalpath11appsrv11-8020 
 publicpath12internalpath12appsrv12-8020 
 } 
  
  Always set X-Forwarded-For for transparency... 
 if {[HTTP::header exists X-Forwarded-For]} { 
 HTTP::header replace X-Forwarded-For [IP::client_addr] 
 } else { 
 HTTP::header insert X-Forwarded-For [IP::client_addr] 
 } 
  
  Set X-Store-Id 
  
     set STOREID Non-Store 
 set ip [IP::client_addr] 
  
  Setting the ip for testing purposes because dev workstation does not belong to store-network 😉 
 set ip 172.20.15.32 
  
     set net_start 172.20.0.0 
 set net_maskbit 14 
 set net_mask 0.3.255.224 
 set net_segment_bit 5 
  
 if { [IP::addr $ip/$net_maskbit equals $net_start] }{  
 scan [IP::addr $ip mask $net_mask] "%d.%d.%d.%d" a b c d  
 set STOREID [expr ( {$a} << 24 | {$b} << 16 | {$c} << 8 | {$d} ) >> {$net_segment_bit} ]  
 if {[HTTP::header exists X-Store-id]} { 
 HTTP::header replace X-Store-Id $STOREID 
 } else { 
 HTTP::header insert X-Store-Id $STOREID 
 } 
 }  
  
     set net_start 10.192.0.0 
 set net_maskbit 10 
     set net_mask 0.63.254.0 
 set net_segment_bit 9 
  
 if { [IP::addr $ip/$net_maskbit equals $net_start] }{  
 scan [IP::addr $ip mask $net_mask] "%d.%d.%d.%d" a b c d  
 set STOREID [expr ( {$a} << 24 | {$b} << 16 | {$c} << 8 | {$d} ) >> {$net_segment_bit} ]  
 if {[HTTP::header exists X-Store-id]} { 
 HTTP::header replace X-Store-Id $STOREID 
 } else { 
 HTTP::header insert X-Store-Id $STOREID 
 }  
 }  
  
  We're going to balance (otherwisethe request will be balanced to the default pool of the virtual server)... 
 set requesteduri [HTTP::uri] 
 foreach { publicpath publicpath srvpool } $publications { 
 if { ([HTTP::uri] starts_with "/$publicpath\/") or ([HTTP::uri] equals "/$publicpath")  } { 
 set internalUri [regsub -nocase "/$publicpath/*" [HTTP::uri] ""] 
 HTTP::uri "/$publicpath/$internalUri" 
 set publication $publicpath 
 pool $srvpool 
 } 
 } 
 } 
  
 when HTTP_REQUEST_SEND { 
   log local0. "Source-IP: [IP::client_addr], Source-Port: [client_port], URI: $requesteduri, Publication: $publication, VKST: $STOREID, SelectedPoolAndNode: [LB::server]" 
 } 
  
 when LB_FAILED { 
 HTTP::respond 200 content {  
   
   
 Apology Page 
   
   
 We are sorry, but the site you are looking for is temporarily out of service
  
 If you feel you have reached this page in error, please try again.  
   
   
 } 
 log local0. "Balancing failed. Publication: $publication" 
 } 
  
 

[JRahm]

A couple things come to mind. Move the math into CLIENT_ACCEPTED since that occurs only once during the session, the scan & expression alone took 130000 cycles in a quick test I just ran. Also, unless you are eventually going to derive the net values dynamically, just use them directly and avoid setting the variables (net_start, net_mask, net_segment_mask). Make your next net_start if an elseif so it is not checked if the first is evaluated.

[Nat_Thirasuttakorn]

Hi Sascha, you may try this...(based on citizen_elah idea) timing on when RULE_INIT { Set publications Syntax: publicpath internalpath pool set static::publications { publicpath1 internalpath1 appsrv1-8020 publicpath2 internalpath2 appsrv2-8020 publicpath3 internalpath3 appsrv3-8020 publicpath4 internalpath4 appsrv4-8020 publicpath5 internalpath5 appsrv5-8020 publicpath6 internalpath6 appsrv6-8020 publicpath7 internalpath7 appsrv7-8020 publicpath8 internalpath8 appsrv8-8020 publicpath9 internalpath9 appsrv9-8020 publicpath10 internalpath10 appsrv10-8020 publicpath11 internalpath11 appsrv11-8020 publicpath12 internalpath12 appsrv12-8020 } } when CLIENT_ACCEPTED { Set default publication and pool set publication vkstintranet pool vkstintranet Set X-Store-Id set STOREID Non-Store set ip [IP::client_addr] Setting the ip for testing purposes because dev workstation does not belong to store-network 😉 set ip 172.20.15.32 set net_start 172.20.0.0 set net_maskbit 14 set net_mask 0.3.255.224 set net_segment_bit 5 if { [IP::addr $ip/$net_maskbit equals $net_start] }{ scan [IP::addr $ip mask $net_mask] "%d.%d.%d.%d" a b c d set STOREID [expr ( {$a} << 24 | {$b} << 16 | {$c} << 8 | {$d} ) >> {$net_segment_bit} ] } set net_start 10.192.0.0 set net_maskbit 10 set net_mask 0.63.254.0 set net_segment_bit 9 if { [IP::addr $ip/$net_maskbit equals $net_start] }{ scan [IP::addr $ip mask $net_mask] "%d.%d.%d.%d" a b c d set STOREID [expr ( {$a} << 24 | {$b} << 16 | {$c} << 8 | {$d} ) >> {$net_segment_bit} ] } } when HTTP_REQUEST { Always set X-Forwarded-For for transparency... if {[HTTP::header exists X-Forwarded-For]} { HTTP::header replace X-Forwarded-For [IP::client_addr] } else { HTTP::header insert X-Forwarded-For [IP::client_addr] } if {[HTTP::header exists X-Store-id] and $STOREID ne "Non-Store" } { HTTP::header replace X-Store-Id $STOREID } else { HTTP::header insert X-Store-Id $STOREID } We're going to balance (otherwisethe request will be balanced to the default pool of the virtual server)... set requesteduri [HTTP::uri] foreach { publicpath publicpath srvpool } $static::publications { if { ([HTTP::uri] starts_with "/$publicpath\/") or ([HTTP::uri] equals "/$publicpath") } { set internalUri [regsub -nocase "/$publicpath/*" [HTTP::uri] ""] HTTP::uri "/$publicpath/$internalUri" set publication $publicpath pool $srvpool } } } when HTTP_REQUEST_SEND { log local0. "Source-IP: [IP::client_addr], Source-Port: [client_port], URI: $requesteduri, Publication: $publication, VKST: $STOREID, SelectedPoolAndNode: [LB::server]" } when LB_FAILED { HTTP::respond 200 content { Apology Page We are sorry, but the site you are looking for is temporarily out of service If you feel you have reached this page in error, please try again. } log local0. "Balancing failed. Publication: $publication" } not tested... Nat

[Nat_Thirasuttakorn]

and if you want something that easily to scale ... try this... (if you dont plan to have more store subnet, probably dont need this... timing on when RULE_INIT { set static::list_net_start "172.20.0.0 10.192.0.0" array set static::array_net_maskbit { "172.20.0.0" 14 "10.192.0.0" 10 } array set static::array_net_mask { "172.20.0.0" "0.3.255.224" "10.192.0.0" "0.63.254.0" } array set static::array_net_segment_bit { "172.20.0.0" 5 "10.192.0.0" 9 } } when CLIENT_ACCEPTED { set ip 10.192.18.0 foreach net_start $static::list_net_start { if { [IP::addr $ip/$static::array_net_maskbit($net_start) equals $net_start] }{ log local0. "net $net_start found." scan [IP::addr $ip mask $static::array_net_mask($net_start)] "%d.%d.%d.%d" a b c d set VKSTID [expr ( {$a} << 24 | {$b} << 16 | {$c} << 8 | {$d} ) >> {$static::array_net_segment_bit($net_start)} ] log local0. "VKSTID = $VKSTID" } } } actually, you already use foreach with publications variable. this is same. (I just prefer to use array than list)