Excluding some urls for the STREAM conversion

Hi Gerardo,

 

 

I assume the issue is with the stream profile rewriting references within the page content from http:// to https://. I expect the app isn't redirecting with a 30x to google or other external sites.

 

 

You can use a negative lookbehind regex to specify the search string in the stream expression. There is an example on the STREAM::enable wiki page (Click here). I've added a few more examples to the STREAM::expression page (Click here) as well.

 

 

 

STREAM::expression {@http://(?!(?:www\.)?amazon\.com|google\.com|ebay\.com)@https://@}

 

 

Would match the bolded sections for these test cases:

 

 

http://example.com

 

http://google.com

 

http://www.amazon.com

 

http://amazon.com

 

http://ebay.com

 

http://test.com

 

 

I'm not sure how much drain this would add on CPU and/or memory. If you can positively define what strings you want to rewrite with a simpler regex, it would be less resource intensive.

 

 

Also, it would be a good idea to disable the stream filter using STREAM::disable by default and only enable it when the content-type header value matches your test. This ensures that the stream filter isn't applied to subsequent HTTP responses on the same TCP connection. There is an example of this on most of the stream wiki pages.

 

 

Aaron

I guess it will be better to have the inclusion stream instead of exclusion.

 

Could you assist me with that?

 

 

I want to convert only calls in the page that make reference to whatever.mycompany.com

 

Whatever may be any number of names including one, two

 

Example

 

one.one.mycompany.com

 

one.two.mycompany.com

 

two.one.mycompany.com

 

three.mycompany.com

 

regional.london.mycompany.com

 

london.mycompany.com

 

africa.mycompany.com

 

new.uk.mycompany.com

 

... (you get the point)

 

mycompany.com

 

 

 

 

 

This is the one that I use

 

when HTTP_RESPONSE {

 

 

Disable the stream filter by default

 

STREAM::disable

 

 

Check if response type is text

 

if {[HTTP::header value Content-Type] contains "text"}{

 

 

Replace any http:// instance with https:// only if the original string is http://*example.com:

 

STREAM::expression {@http://(?:.*?example\.com)@https://@}

 

 

Enable the stream filter for this response only

 

STREAM::enable

 

}

 

}

 

 

All the https have https:///folderone/image.gif instead of https://example.com/folderone/image.gif

 

 

How to fix this?

 

This is what I got in the log file

 

 

TCL error: Rule nameoftherule HTTP_RESPONSE - Illegal argument line 9 invoked from within STREAM::expression {http://?:.*?example.com}

I would have thought you could use a stream expression with a single token. Here is a more tested example with the unnecessary backreference in the regex removed:

 
 when HTTP_RESPONSE {  
    
      Disable the stream filter by default  
     STREAM::disable  
    
      Check if response type is text  
     if {[HTTP::header value Content-Type] contains "text"}{  
    
         Match any http:// instance and replace it with nothing 
        STREAM::expression {@http://.*?example\.com@@} 
    
         Enable the stream filter for this response only  
        STREAM::enable  
     }  
  }   
  when STREAM_MATCHED {  
     log local0. "[IP::client_addr]:[TCP::local_port]: matched: [STREAM::match], replaced with: [string map {http:// https://} [STREAM::match]]"  
     STREAM::replace "[string map {http:// https://} [STREAM::match]]"  
    
  } 
 

And the output:

Rule stream_expression_rule : 10.0.0.1:3413: matched: http://test.example.com, replaced with: https://test.example.com

Rule stream_expression_rule : 10.0.0.1:3413: matched: http://example.com, replaced with: https://example.com

Aaron

That worked great, thanks

 

Now I need more help

 

With more modifications to the same iRule to exclude some stuff.

 

 

 

Where can I find more information about the Regular Expressions and their validation?

 

I'm sorry it did not work.

 

 

If you include a url in your test with the name somethingelse.com it will convert that one too not only *.example.com

 

 

 

Here are some link to learn more about regular expressions

 

 

Click here

 

Click here

 

Click here

 

 

I think hoolio rule shouldn't match anything else, maybe you've done some typo.

 

 

Do you try the exact same iRule or did you update it ? if yes please post your code

This is the iRule that I have, I just changed the name of the url to the one for the company.

 

 

This is the version BIG-IP 9.2.4 Build 13.1

 

 

 

when HTTP_RESPONSE {

 

if { not ([matchclass [IP::client_addr] equals $::internal_net])}{

 

 

 

Enable debug logging to /var/log/ltm? 1=yes, 0=no

 

set debug 1

 

STREAM::disable

 

If the response content type is text, configure the default stream profile to replace http:// with https://

 

if {[string tolower [HTTP::header value Content-Type]] contains "text"}{

 

STREAM::expression {@http://.*?example\.com@@}

 

STREAM::enable

 

}

 

Check if the response is a redirect (3xx)

 

if {[HTTP::is_redirect]}{

 

 

if {$debug}{log local0. "Original Location: [HTTP::header value Location], Updated Location: [string map {http: https:} [HTTP::header value Location]]"}

 

Use 'string map' to replace http: with https:

 

HTTP::header replace Location [string map {http: https:} [HTTP::header value Location]]

 

}

 

}

 

}

 

 

when STREAM_MATCHED {

 

log local0. "[IP::client_addr]:[TCP::local_port]: matched: [STREAM::match], replaced with: [string map {http:// https://} [STREAM::match]]"

 

STREAM::replace "[string map {http:// https://} [STREAM::match]]"

 

 

}

The stream portion of the rule should be working fine to only rewrite the http:// references to https:// for *.example.com. The issue is likely to be there isn't any checking on 30x redirects. If you change this line:

 

 

if {[HTTP::is_redirect]}{

 

 

To:

 

 

if {[HTTP::is_redirect] && [string match -nocase "http://*.example.com*" [HTTP::header value Location]]}{

 

 

You can restrict the Location header rewriting to redirects which match http://*.example.com*

 

 

If this doesn't work, try adding logging to the rule, reproduce the issue and then repost your current rule with the logging and the log output from /var/log/ltm. If you have any problems debugging the rule, let us know.

 

 

Aaron