Excluding some urls for the STREAM conversion

Here are some examples of non working things

 

image.weather.com, w3.org, plumtree.com are not in part of my domain

 

 

Is there any way to check row by row in the code instead of chuncks of code?

 

 

 

Rule stageiRuleGerardopart3 STREAM_MATCHED: 161.51.3.239:3366: matched: http://image.weather.com/web/services/oap/magnet_hgeneric.jpg WIDTH=270 HEIGHT=175TRTD VALIGN=TOPTABLE BORDER=0 CELLPADDING=0 CELLSPACING=0 WIDTH=270 BACKGROUND=TRTDIMG SRC=http://image.weather.com/web/blank.gif WIDTH=1 BORDER=0 HEIGHT=1 alt=/TDTDIMG SRC=http://image.weather.com/web/blank.gif WIDTH=96 BORDER=0 HEIGHT=1 alt=/TDTDIMG SRC=http://image.weather.com/web/blank.gif WIDTH=168 BORDER=0 HEIGHT=1 alt=/TD/TRTRTDIMG SRC=http://image.weather.com/web/blank.gif WIDTH=1 BORDER=0 HEIGHT=20 alt=/TDTD ALIGN=CENTER COLSPAN=2 VALIGN=MIDDLEIMG SRC=http://image.weather.com/web/blank.gif WIDTH=1 BORDER=0 HEIGHT=1 alt=A title=As reported at Houston Wb City, TX. Jul 24 12:05 p.m. CT target=blank HREF=http://www.weather.com/weather/local/USTX0617?cm_ven=WOWs_voapcm_cat=Genericcm_ite=linkcm_pla=Cityname CLASS=GENXHcityInfoBHouston, TX

 

 

Rule stageiRuleGerardopart3 STREAM_MATCHED: 161.51.3.239:3366: matched: http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd span xmlns:pt=http://www.plumtree.com/xmlschemas/ptui/ script type=text/javascriptif typeofMyKbr==object MyKbr.styleSheetFix/script script type=text/javascript function MyKbr_footerOpenContactInfoevent { if !MyKbr_footerCancelCloseContactInfo { var e=document.getElementByIdMyKbr_footerContactInfo e.style.left=event.clientXdocument.body.scrollLeft10 e.style.top=event.clientYdocument.body.scrollTop-e.offsetHeight-10 e.style.visibility=visible } } function MyKbr_footerCancelCloseContactInfo { if MyKbr_footerCloseContactInfo.timerHandle { window.clearTimeoutMyKbr_footerCloseContactInfo.timerHandle MyKbr_footerCloseContactInfo.timerHandle=null return true }

 

 

Rule stageiRuleGerardopart3 STREAM_MATCHED: 161.51.3.239:3550: matched: http://www.plumtree.com/xmlschemas/ptui/ script language=javascriptdocument.cookie=mykbrexp=/script table style=BACKGROUND-COLOR: white border=0 cellpadding=0 cellspacing=0 width=100 height=40 tr td width=150 valign=topdiv style=width:150height:50overflow:hidden img style=position:relativeleft:-20top:-13 id=imgLogo src=http://static.stage.regional.mykbr.com, replaced with: https://www.plumtree.com/xmlschemas/ptui/ script language=javascriptdocument.cookie=mykbrexp=/script table style=BACKGROUND-COLOR: white border=0 cellpadding=0 cellspacing=0 width=100 height=40 tr td width=150 valign=topdiv style=width:150height:50overflow:hidden img style=position:relativeleft:-20top:-13 id=imgLogo src=https://static.stage.regional.mykbr.com

 

OK, I think someone needs to help me here

 

 

This is the new iRule and it works following the first example here

 

http://devcentral.f5.com/wiki/default.aspx/iRules/STREAM__expression.html

 

 

I don't know very well regular expressions but this is my solution.

 

 

when HTTP_RESPONSE {

 

if { not ([matchclass [IP::client_addr] equals $::internal_net])}{

 

set debug 1

 

STREAM::disable

 

STREAM::expression {@http://stage.example.com@https://stage.example.com@ @http://static.regional.example.com@https://static.regional.example.com@ @http://static.stage.regional.example.com@https://static.stage.regional.example.com@ @http://static.example.com@https://static.example.com@ }

 

STREAM::enable

 

}

 

}

 

 

 

If you think you can improve it plese do it.

 

One importan thing here is that I cannot convert all the *.example I need to have a set of domains with example not all of them.

 

 

For example stage.server1.example.com should be excluded and maybe others.

 

 

 

That's interesting... It looks like the ? in the regex might not be performing a lazy match on the string being searched.

In a quick test, the ? works for regexp to restrict the match to the shortest possible string:

 
 when RULE_INIT { 
    log local0. "[regexp -inline {test.*?example\.com} {test1.example.com text text test2.example.com}]" 
 } 
 

Log output:

Rule : test1.example.com

Likewise, using the last rule I posted worked for a test below:

 
 when HTTP_RESPONSE {  
    
     Disable the stream filter by default  
    STREAM::disable  
    
     Check if response type is text 
    if {[HTTP::header value Content-Type] contains "text"}{  
  
       log local0. "[IP::client_addr]:[TCP::client_port]: Setting stream expression, enabling filter"   
        Replace any http:// instance with https:// only if the original string is http://*example.com:  
       STREAM::expression {@http://.*?example\.com@@} 
  
        Enable the stream filter for this response only  
       STREAM::enable 
    } 
 }   
 when STREAM_MATCHED {  
    log local0. "[IP::client_addr]:[TCP::local_port]: matched: [STREAM::match], replaced with: [string map {http:// HTTPS://} [STREAM::match]]"  
    STREAM::replace "[string map {http:// HTTPS://} [STREAM::match]]"  
 } 
 

Here is the log output for a request which was rewritten:

: 10.1.1.1:4683: matched: http://test.example.com, replaced with: HTTPS://test.example.com

: 10.1.1.1:4683: matched: http://example.com, replaced with: HTTPS://example.com

And here is the updated response content:

HTTPS://test.example.com

HTTPS://example.com

https://example.com

http://nochange.com

To compare, this is a request direct to the server:

http://test.example.com

http://example.com

https://example.com

http://nochange.com

I'm not sure why the rule isn't working when you're testing against your application. I would suggest opening a case with F5 Support referencing this post, as the example is working on a simple test in 9.4.4 and should work for your scenario.

On your second point, if you need to only match some subdomains but not all, you could either modify the regex in the stream expression, or add logic to the STREAM_MATCHED event. For the second option, you can check if the matched string is in a list or datagroup of subdomains to not rewrite. In that case, just don't call STREAM::replace.

Aaron

Thinking about this again, I'm guessing the reason you were seeing strings being matched which were much longer than a single domain was the regex you were testing with was wrong.

Here is an example that should work to allow you to check all http://*.example.com strings but rewrote only those you want to.

    
    when RULE_INIT {    
        
        List of *.example.com subdomains that should not be rewritten (in lowercase)    
       set ::subddomains_to_preserve [    
          "keep1.example.com" \    
          "keep2.example.com" \    
          "keep3.example.com" \    
       ]    
    }    
    when HTTP_RESPONSE {      
        
        Disable the stream filter by default      
       STREAM::disable      
        
        Check if response type is text     
       if {[HTTP::header value Content-Type] contains "text"}{      
        
          log local0. "[IP::client_addr]:[TCP::client_port]: Setting stream expression, enabling filter"       
           Replace any http:// instance with https:// only if the original string is http://*example.com:      
          STREAM::expression {@http://.*?example\.com@}     
        
           Enable the stream filter for this response only      
          STREAM::enable     
       }     
    }       
    when STREAM_MATCHED {    
        
        Check if the current string matched by the stream filter is part of the list of subdomains to preserve.    
       if {[matchclass [string tolower [STREAM::match]] equals $::subddomains_to_preserve]}{    
          log local0. "[IP::client_addr]:[TCP::local_port]: found [STREAM::match], but did not modify it."      
       } else [    
          log local0. "[IP::client_addr]:[TCP::local_port]: found [STREAM::match], replaced with: [string map {http:// HTTPS://} [STREAM::match]]"      
          STREAM::replace "[string map {http:// https://} [STREAM::match]]"      
       }    
    }    
    

You could also change the list in RULE_INIT from subdomains to preserve to those you want to rewrite to https. The list could also be replaced with a separate class.

Edit: you'll also want to use an HTTP profile with response chunking set to rechunk.

SOL6422: Using the Stream profile with HTTP traffic can prevent the client from displaying all of the data

https://support.f5.com/kb/en-us/solutions/public/6000/400/sol6422.html

Aaron

Hi everybody, I been copying and pasting the last irule shown here, because I had a similar issue. First I had to modify that irule becouse gave to me some problems whit the syntax and other stuff. I'm new on f5 so I starting to google it how to fix it and finally I got this:

when RULE_INIT {    

    Lista de los domininios que NO se reemplazaran 
  set static::listado [ list /http://www.fapesp.br ]
    }

when HTTP_RESPONSE {      

    Desabilitar el filtro stream 
   STREAM::disable      

    Check if response type is text     
   if {[HTTP::header value Content-Type] contains "text"}{      

      log local0. "[IP::client_addr]:[TCP::client_port]: Setting stream expression, enabling filter"       
       Remplazar cualquier http:// por https://      
      STREAM::expression {@http://*@}     

       Habilitar el filtro stream solo si corresponde
      STREAM::enable     
   }     
}       
when STREAM_MATCHED {    

    Revisar if el string actual obtenido por el filtro stream esta en la lista de dominios a conservar
   if { [lsearch -glob $static::listado [string tolower [STREAM::match]] ] } {    
      log local0. "[IP::client_addr]:[TCP::local_port]: found [STREAM::match], but did not modify it."      
   } else {   
      log local0. "[IP::client_addr]:[TCP::local_port]: found [STREAM::match], replaced with: [string map {http:// HTTPS://} [STREAM::match]]"      
      STREAM::replace "[string map {http:// https://} [STREAM::match]]"      
   }    
}

This rule does not works and show this error: " Rule /Common/iRule_Scielo_EXCEPTIONS : 192.168.210.36:19119: found http://, but did not modify it"

So where is my problem, I only want excluding some urls from stream conversion. Thank