Fetching content from external source

Aron,

 

 

Can you explain what are you trying to do by making an out of band request to a pool? Are you determining if the client request is valid? Or something else?

 

 

Actually I want to do a lookup from an external HTTP (RESTful) server to get a Mobile Phone Number (MDN) from the IP of the requesting client e.g. "/[IP::client_addr]" these mobile phones route through the F5 to the internet at large.

 

 

Yes it is an out-of-band socket connection to a 3rd host - but in the event the 3rd host is down or it fails to provide a valid MDN then the out-of-band operation does nothing and the out-of-band logic should be ignored.

 

 

Future Work: If I couldn't look up the MDN I might re-try the out-of-band operation again once each 30 seconds as long as the connection is up.

 

 

Future Work: Once I have an MDN (if and only if I have one) then I might do some HTML injection on the response or a redirect.

 

 

Here is some proposed code - hopefully it will work any advice?

 

 

 

     
      when CLIENT_ACCEPTED {      
         flag as new request needing lookup      
        set lookup 1      
         set flag to indicate if we had a valid lookup.      
        set keyfound 0         
      }       
      when HTTP_REQUEST {      
        if {$lookup == 1} {      
           we are doing a 3rd host out-of-band lookup     
           save original request      
          set req [HTTP::request]      
      
           inject a lookup URI in place of original request      
          HTTP::uri "/[IP::client_addr]"      
      
           set pool to lookup server pool called "mdn_lookup"      
           hopefully the pelow "pool mdn_lookup" is not sticky    
           such that the later HTTP::retry $req done in the   
           HTTP_RESPONSE_DATA will not go back to this pool, I    
           can not set it to another "normal" pool after this    
           out-of-band processing is done as the requests are    
           going to the internet at large (not servers under    
           my control).    
          pool mdn_lookup      
        }        
      }      
      when HTTP_RESPONSE {      
        if {$lookup == 1 }{      
           collect first response (from lookup server) only      
          HTTP::collect      
    
          if {[HTTP::status] == 200} {      
         200 OK we got an IP lookup from the RESTful server      
         we have to have a valid MDN or MobileNumber, all other         
         responses imply no lookup.        
              
         collect first response (from lookup server) only        
            HTTP::collect $clength      
                
             valid lookup we have a MDN and can do something with it        
            set keyfound 1        
          }      
        }  
         normal path for NOT out-of-band responses  
      }      
      when HTTP_RESPONSE_DATA {      
         Once the lookup is done there is NO pool to go back to      
         as the clients are mobile phones going out to the internet        
            
        if {$keyfound == 1} {      
      set mdn [HTTP::payload]      
      log local0. "mdb_lookup gives <<$mdn>>"          
       we want to actually do something with it later but       
       this iRULE is a proof of concept.  Typical we would      
       alter the true response based on the mdn we just set.      
       If we never get a valid MDN we the request will       
       proceed as if the out-of-band lookup never happened.      
        }  else {      
      log local0. "mdb_lookup gives NULL"      
        }      
        HTTP::retry $req        
              
         NO ERROR CHECKING YET ....      
         if mdn_lookup is dead/non-existant I just want to      
         process things as if there was no out-of_band code      
            
         re-set flag so that subsequent response to re-tried       
         request from real server is not processed as an MDN      
         lookup - eventually I might put in retry every 15       
         second logic until we get a valid lookup      
        set lookup 0      
      }      
     

Fetching content from external source, seems to be a bit buggy, at leas in my use case

where I have a pair of load balanced servers that share the same nodes. For some reason

selecting a pool doesn't seem to force a particular port.

Question 1, Why do I have to define virtual servers (it will not work at all if I do not),

from the examples I though I only needed to reference pools ?

Question 2, Why does it only work http://10.0.185.150:8002/prox_evdo/auth.gif?rnd12345,

or why doesn't the right OOB port, 8002, get selected on the fetch from

http://10.0.185.150/prox_evdo/auth.gif?rnd12345?

  
   Based on iRULEs examples ....  
   SEE: http://devcentral.f5.com/Default.aspx?tabid=63&articleType=ArticleView&articleId=105  
   SEE: http://devcentral.f5.com/Default.aspx?tabid=53&view=topic&forumid=5&postid=14001  
   SEE: http://devcentral.f5.com/Wiki/default.aspx/iRules/OpenSSO_authentication.html  
    
    
   I have two virtual servers (that share the same LB nodes) with this same rule attached  
   A) mdn_track 10.0.185.150:80                nodes   10.0.180.3  and   10.0.180.5  
   B) mdn_lookup 10.0.185.150:8002             nodes   10.0.180.3  and   10.0.180.5  
    
   *) Note, mdn_track is a web server and /prox_evdo/auth.gif returns a 1x1 gif  
   *) Note, mdn_track always returns "808111222  "  
   *) Note, on the F5 we have self IPs for 10.0.180.10&11 and 10.0.185.10&11  
    
   THIS REQ WORKS (mdn_lookup goes to port 8002, mdn_track goes to port 80)  
   http://10.0.185.150:8002/prox_evdo/auth.gif?rnd12345  
    
    : OOB mdn_lookup host <10.0.185.150:8002> mod-uri  LB::server   
    : OOB mdn_lookup: response STAT <200> lbsvr  stat up conlen <10>  
    : OOB mdn_lookup: STAT <200> VALID payload ... <8081112222  >, retry orig  
    : new req   
    : mdn_track host <10.0.185.150:8002> orig-uri  LB::server   
    : mdn_track: response STAT <200> lbsvr  stat up conlen <44>  
    
   THIS REQ FAILS (mdn_lookup goes to port 80, mdn_track never called) why doesn't "pool mdn_lookup" select the 8002 port  
   http://10.0.185.150/prox_evdo/auth.gif?rnd12345  
    
    : OOB mdn_lookup host <10.0.185.150> mod-uri  LB::server   
    : OOB mdn_lookup: response STAT <404> lbsvr  stat up conlen <271>  
    : OOB mdn_lookup: STAT <404> REJECT payload ... <  
    
   Question 1, Why do I have to define virtual servers (it will not work at all if I do not),  
                from the examples I though I only needed to reference pools ?  
    
   Question 2, Why does it only work http://10.0.185.150:8002/prox_evdo/auth.gif?rnd12345,  
                or why doesn't the right OOB port, 8002, get selected on the fetch from  
                http://10.0.185.150/prox_evdo/auth.gif?rnd12345?  
    
  when CLIENT_ACCEPTED {  
     set lookup 1  
     set valid_string "808"  
  }  
  when HTTP_REQUEST {  
       if { [HTTP::version] eq "1.1" } {  
          if { [HTTP::header is_keepalive] } {  
              HTTP::header replace "Connection" "Keep-Alive"  
          }  
          HTTP::version "1.0"  
      }  
      if {$lookup == 1} {  
          set def_server [LB::server]  
          set LB_request [HTTP::request]  
           inject lookup URI in place of original request;  
          HTTP::uri "/[IP::client_addr]"  
           the below never seems to update the port only works if initial port is 8002, it fails  
           to update the port to 8002 if the original request is 80  
          pool mdn_lookup  
          set oob_server [LB::server]  
          log local0. "OOB mdn_lookup host <[HTTP::host]> mod-uri <[HTTP::uri]> LB::server <$oob_server>"  
      } else {  
          log local0. "mdn_track host <[HTTP::host]> orig-uri <[HTTP::uri]> LB::server <$def_server>"  
          pool mdn_track  
      }  
  }  
  when HTTP_RESPONSE {  
    if {$lookup == 1} {  
      log local0. "OOB mdn_lookup: response STAT <[HTTP::status]> lbsvr <[LB::server]> stat [LB::status] conlen <[HTTP::header Content-Length]>"  
      if {[HTTP::header exists Content-Length] && \  
         ([HTTP::header Content-Length] < 1048576)}{  
         set clength [HTTP::header Content-Length]  
      } else {  
         set clength 1048576  
      }  
      HTTP::collect $clength  
    } else {  
      log local0. "mdn_track: response STAT <[HTTP::status]> lbsvr <[LB::server]> stat [LB::status] conlen <[HTTP::header Content-Length]>"  
    }  
  }  
  when LB_FAILED {  
      log local0.debug "LB failed for: [LB::server]"  
      log local0.debug "LB status: [LB::status]"  
  }  
  when HTTP_RESPONSE_DATA {  
    if {[HTTP::payload] contains $valid_string}{  
      log local0. "OOB mdn_lookup: STAT <[HTTP::status]> VALID payload ... <[HTTP::payload]>, retry orig"  
       the below always seems to update the port to 80 as expected, evenif the original port was 8002  
      pool mdn_track  
      log local0. "new req <$LB_request>"  
      HTTP::retry $LB_request  
    } else {  
      log local0. "OOB mdn_lookup: STAT <[HTTP::status]> REJECT payload ... <[HTTP::payload]>"  
      reject  
    }  
    set lookup 0  
  }  
    
  

Hi Jon,

 

 

How did you have the VIP defined when were you trying to add the iRule to a 0.0.0.0 VIP with a pool? Was it a Performance / FastL4 VIP? If so, can you try with a standard IP 0.0.0.0 VIP instead?

 

 

Also, can you move this line from HTTP_REQUEST to CLIENT_ACCEPTED and retest?

 

 

set def_server [LB::server]

 

 

LB::server will return the currently selected pool--not necessarily the VIP's default pool. The value is only reset to the default pool value on a new TCP connection.

 

 

Thanks,

 

Aaron

Hi Jon,

> How did you have the VIP defined when were you trying to add the iRule to a 0.0.0.0 VIP with a

> pool? Was it a Performance / FastL4 VIP? If so, can you try with a standard IP 0.0.0.0 VIP instead?

1. I have a main virtual server "vs_main" that gets all non-http traffic it is (performance layer-4)

no profiles but a single iRULE showing debug if needed on just 'connections'

2. I have a focused virtual server "vs_main_http" that gets port 80 traffic (but in the above posting

NOT 8002 traffic) it is (type: Standard - with a HTTP/httpclass profile

-- Perhaps when I select the virtual server for :8080 it goes to vs_main (and there is NO

HTTP/httpclass profile) this might be my entire issue.

It also might explain why "I needed to create virtual servers with HTTP/httpclass profile's"

on top of my LB node pairs to get anything to work at all.

> Also, can you move this line from HTTP_REQUEST to CLIENT_ACCEPTED and retest?

> set def_server [LB::server]

 I retested (moving the def_server line) no luck - BUT the key was to enable  
 "port translation" on the VIP - thanks to your pointer (different post).  
  
 Obviously I don't fully grasp all the concepts of the F5 there but I am learning 

> LB::server will return the currently selected pool--not necessarily the VIP's default pool.

> The value is only reset to the default pool value on a new TCP connection.

Interesting, I still have one question how does one "force" the VIP's default pool

Thanks in Advance

Jon

Hi Jon,

 

 

Let me know how the testing goes with the LB::server command moved to the CLIENT_ACCEPTED event. That is the way to force the use of the VIP's default pool--save the name of the default pool before it's been changed on the TCP connection using the LB::server command in the CLIENT_ACCEPTED event and select it using 'pool $def_pool' in the rest of the iRule.

 

 

Aaron

We route requests to destination servers based upon a value they have in an LDAP server. In order to do this with F5, we created a lookup script which will return the pertinent values in a header. The iRule then parses this header and will route based upon which application the web service is intended for. Upon testing it, we figured out that the default HTTP:retry was not sending the original payload. In order to fix this, we had to append the original payload to the original headers. I have pasted the iRule below, has anyone else seen this issue or know of another way to get around it?

 

 

when CLIENT_ACCEPTED {

 

set flag to control logical flow. 1 means lookup is pending.

 

set lookup 1

 

}

 

 

when HTTP_REQUEST {

 

HTTP::version "1.0"

 

set param "ID"

 

set search_string "Portal"

 

set default_pod "p_webservice_default"

 

 

if {$lookup eq 1} {

 

set OID [HTTP::header $ID]

 

 

if {OID ne "" } {

 

log local0. "LookUp Log: ID found: $OID"

 

save the payload, trigger collection for up to 1MB of data

 

if {[HTTP::header exists "Content-Length"] && [HTTP::header "Content-Length"] <= 1048576}{

 

set content_length [HTTP::header "Content-Length"]

 

} else {

 

decide what to do here, throw soap fault maybe?

 

drop

 

}

 

Check if $content_length has been set and is not set to 0

 

if { [info exists content_length] && $content_length > 0} {

 

HTTP::collect $content_length

 

}

 

save original request headers

 

set origRequest [HTTP::request]

 

log local0. "LookUp Log: Original Request is: $origRequest"

 

HTTP::header replace "ID" $OID

 

inject lookup URI in place of original request

 

HTTP::uri "/lookup.php"

 

Route to the lookup service

 

node x.x.x.x 80

 

}

 

} else {

 

try to route to appropriate pod

 

if {![catch {pool p_webservice_$resolved_pod}]} {

 

log local0. "Web Service Log: Routing finally to p_webservice_$resolved_pod"

 

} else {

 

let the request go to default pod

 

log local0. "Web Service Log: Routing to default pod (reason was $resolved_pod)"

 

pool $default_pod

 

}

 

}

 

}

 

when HTTP_REQUEST_DATA {

 

if {($lookup eq 1) and ($OID ne "")}{

 

save the payload

 

set originalPayload [HTTP::payload]

 

HTTP::release

 

}

 

}

 

when HTTP_RESPONSE {

 

HTTP::version "1.0"

 

if {($lookup eq 1) and ($OID ne "") }{

 

set lookup 0

 

log local0. "LookUp Log: Response from lookup received, LBGroup: [HTTP::header LBGroup], OID: [HTTP::header OID_Sent]"

 

set full_header [HTTP::header LBGroup]

 

switch -glob $full_header {

 

"ERROR" {

 

ERROR CONDITIONS AND RESPONSES

 

}

 

default {

 

set resolved_pod [findstr $full_header $search_string 11 ";"]

 

log local0. "LookUp Log: Routing to resolved pod: $resolved_pod, and length is [string length $resolved_pod]"

 

Make sure this OrgOID has a POD for this product ($search string)

 

if {[string length $resolved_pod] > 0} {

 

verify pool exists and has members

 

if {![catch [pool p_webservice_$resolved_pod]]}{

 

Retry with the original request

 

log local0. "LookUp before retry Log: Original Payload is: $originalPayload"

 

append origRequest $originalPayload

 

log local0. "LookUp before retry Log: Request + Payload is: $origRequest"

 

HTTP::retry $origRequest

 

} else {

 

POD is invalid or was not defined in F5 pools

 

log local0. "Lookup Log Critical Error PODNOTFOUNDINF5: OrgID is $ADP_ORGOID has a valid RG $resolved_pod with no F5 pool defined"

 

set soapFaultPodNotFound "Inv...de>Invalid Resource GroupLDAP Search failedWeb Tier F5 Gateway"

 

HTTP::header replace "Content-Type" "text/xml"

 

HTTP::payload replace 0 [HTTP::header value Content-Length] $soapFaultPodNotFound

 

}

 

} else {

 

this is odd ball case where lookup would not return a pod or know error code

 

review if we need it or we can reject request here

 

set resolved_pod "INVALIDCODE"

 

append origRequest $originalPayload

 

HTTP::retry $origRequest

 

}

 

}

 

}

 

}

 

}

 

when HTTP_RESPONSE_DATA {

 

set lookup 0

 

}

 

 

when LB_SELECTED {

 

log local0. "*** Web Service Log: Selected [LB::server name]:[LB::server port]"

 

}

 

HTTP::request only contains the HTTP headers in a request. If you want to retry a request with the original payload you would need to collect the payload and append it to the request you're retrying.

 

 

So what you're seeing is expected.

 

 

Aaron