Check HTTP Response Data for string

Question

Hi there, &nbsp;
&nbsp;We've got an intermittent issue with one of our applications which I'm trying to get to understand a bit better using our F5's and iRules... &nbsp;
&nbsp;Basically, sometimes the application fails to include the head block in the HTML responses. &nbsp;
&nbsp;I want to be able to use an iRule to log every time this happens?&nbsp;
&nbsp;What I'm trying to get is the most economical and reliable way of searching the response data for the given string. 
&nbsp;Is this something that can be achieved using STREAM profiles? &nbsp;
&nbsp;Comments welcome... &nbsp;
&nbsp;Cheers
&nbsp;Gav&nbsp;

gavinw_29074 · Answer

Hi there,  
  
 So I went ahead and am trying to use Stream Expressions to check for the string...  
  
 I've got the iRule compiling the expression and it appears to be setting it ok. However it's making NO impact on the resultant HTML returned to the client...  
  
 Copy of the rule and log file excerpt below...  
  
 Any ideas??? 
  
 Cheers 
 Gavin

Source Control:
Where: $HeadURL: http://subversion.card.co.uk/svn/products/Utilities/F5/iRules/++Common++SplunkHTTP.txt $
Last Modified On: $Date: 2012-02-16 12:27:33 +0000 (Thu, 16 Feb 2012) $
Last Modified By: $Author: gavinw $
Revision: $Revision: 57745 $

when RULE_INIT {

set static::LogInvalidRespDebug 1

}

when HTTP_REQUEST priority 100 {

if {$static::LogInvalidRespDebug &gt; 0} { log local0.info "Processin HTTP Request at Priority 100..." }

if { [HTTP::header exists "X-Requested-With"] } {
if {$static::LogInvalidRespDebug &gt; 0} { log local0.info "X-Requested-With HTTP Header Present. Must be AJAX Request." }
return
} else {
if {$static::LogInvalidRespDebug &gt; 0} { log local0.info "Non AJAX Request. Processing further." }

Only looking for HTML Requests
if { [HTTP::header "Accept"] contains "text/html" } {
if {$static::LogInvalidRespDebug &gt; 0} { log local0.info "Accept Header contains 'text/html'. Recording." }

Prevent the server from sending a compressed response
 remove the compression offerings from the client
HTTP::header remove "Accept-Encoding"

Don't allow response data to be chunked
if { [HTTP::version] eq "1.1" } {

Force downgrade to HTTP 1.0, but still allow keep-alive connections.
 Since HTTP 1.1 is keep-alive by default, and 1.0 isn't,
 we need make sure the headers reflect the keep-alive status.

Check if this is a keep alive connection
if { [HTTP::header is_keepalive] } {
 Replace the connection header value with "Keep-Alive"
HTTP::header replace "Connection" "Keep-Alive"
}

Set server side request version to 1.0
 This forces the server to respond without chunking
HTTP::version "1.0"
}

if {$static::LogInvalidRespDebug &gt; 0} { log local0.info "Storing HTTP::request value" }
set req_data [HTTP::request]
if {$static::LogInvalidRespDebug &gt; 0} { log local0.info "\$req_data = $req_data" }

} else {
if {$static::LogInvalidRespDebug &gt; 0} { log local0.info "Non HTML request. Returning. " }
return
}

}

when HTTP_RESPONSE {

if {$static::LogInvalidRespDebug &gt; 0} { log local0.info "Processing HTTP Response." }

Disable the stream filter by default  
STREAM::disable

Check if response type is text  
if {[HTTP::header value Content-Type] contains "text/html"}{  
if {$static::LogInvalidRespDebug &gt; 0} { log local0.info "HTTP Response Content-Type header contains 'text/html'" }

set lir_stream_find "ACT"
set lir_stream_replace "toe"
set lir_stream_expr "@$lir_stream_find@$lir_stream_replace$lir_stream_find@"
set lir_stream_expr_cmd "STREAM::expression \{\"$lir_stream_expr\"\}"
set lir_stream_enbl_cmd "STREAM::enable"
if { $static::LogInvalidRespDebug &gt; 0 } {
log local0. "\$lir_stream_expr_cmd: $lir_stream_expr_cmd, \$lir_stream_enbl_cmd: $lir_stream_enbl_cmd"
}

Execute the STREAM::expression command.  Use catch to handle any errors. Save the result to $result
if { [catch {eval $lir_stream_expr_cmd} lir_result] } {
 There was an error trying to set the stream expression.
log local0. "Error setting stream expression ($lir_result). Stream profile not set."
} else {
 No error setting the stream expression, so try to enable the stream filter
 Execute the STREAM::enable command.  Use catch to handle any errors. Save the result to $result
if { [catch {eval $lir_stream_enbl_cmd} result] } {
 There was an error trying to enable the stream filter.
log local0. "Error enabling stream filter ($lir_result): $lir_result"
} else {
if { $static::LogInvalidRespDebug &gt; 0 } {
log local0. "Successfully configured and enabled stream filter"
}
}
}

Trigger collection for up to 1MB of data
if {[HTTP::header exists "Content-Length"] &amp;&amp; [HTTP::header "Content-Length"] &lt;= 1048576}{
set content_length [HTTP::header "Content-Length"]
} else {
set content_length 1048576
}

Check if $content_length is not set to 0
if { $content_length &gt; 0} {
HTTP::collect $content_length
}

}
}

when HTTP_RESPONSE_DATA {

do stuff with the payload
set payload [HTTP::payload]
}

when STREAM_MATCHED {

if {$static::LogInvalidRespDebug &gt; 0} { log local0.info "Stream_Matched event triggered" }
log local0. "[IP::client_addr]:[TCP::local_port]: matched: [STREAM::match].]"  
STREAM::replace "[string map {http:// https://} [STREAM::match]]"  
}  
  
  Apr 12 10:23:37 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : Processin HTTP Request at Priority 100... 
 Apr 12 10:23:37 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : Non AJAX Request. Processing further. 
 Apr 12 10:23:37 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : Accept Header contains 'text/html'. Recording. 
 Apr 12 10:23:37 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : Storing HTTP::request value 
 Apr 12 10:23:37 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : $req_data = GET /CMSBackOffice/home HTTP/1.0  Host: 192.168.151.11  Connection: Keep-Alive  Cache-Control: max-age=0  User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.152 Safari/535.19  Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8  Referer: http://192.168.151.11/CMSBackOffice/logon  Accept-Language: en-GB,en-US;q=0.8,en;q=0.6  Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3  Cookie: JSESSIONID=5dd69746a6c26778d7f9ef978282; jsonTable={"1148975717": [1334220684671, 1, 0], "1495321059": [1334221943386, 1, 0, 0, 0], "1833417362": [1334220684691, 1, 0, 0, 0], "2105498430": [1334220684687, 1, 0, 0, 0], "2779550008": [1327417348363, 1, 0, 0, 0], "3714451117": [1334220684675, 1, 0, 0, 3], "4138540245": [1334220684697, 1, 0, 0, 0], "v": 31295000} 
 Apr 12 10:23:38 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : Processing HTTP Response. 
 Apr 12 10:23:38 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : HTTP Response Content-Type header contains 'text/html' 
 Apr 12 10:23:38 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : $lir_stream_expr_cmd: STREAM::expression {"@ACT@toeACT@"}, $lir_stream_enbl_cmd: STREAM::enable 
 Apr 12 10:23:38 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : Successfully configured and enabled stream filter

hooleylist · Answer

Hi Gavin, 
&nbsp;  
&nbsp; Can you make the same request using curl and post the anonymized response headers and content? 
&nbsp;  
&nbsp; As an aside, you can retrieve the payload that the server includes in the packet(s) TMM needs to collect to parse the response headers using [HTTP::payload] in HTTP_RESPONSE (without calling HTTP::collect).  So if the &lt; head &gt; tag is present in the first response packet on successful responses you could potentially log the details for failures in HTTP_RESPONSE without using a stream profile or collecting the payload. 
&nbsp;  
&nbsp; Aaron

gavinw_29074 · Answer

Aaron 
  
 As it happens, i've got it working a slightly different way.  
  
 Comments welcome on possible improvements though 🙂 
  
 Gav 
  
 Copy of my rule is:

when RULE_INIT {

set static::LogInvalidRespDebug 1

}

when HTTP_REQUEST priority 100 {

if {$static::LogInvalidRespDebug &gt; 0} { log local0.info "Processin HTTP Request at Priority 100..." }

Bypass by default
set bypass 1