Check HTTP Response Data for string

Hi there,

So I went ahead and am trying to use Stream Expressions to check for the string...

I've got the iRule compiling the expression and it appears to be setting it ok. However it's making NO impact on the resultant HTML returned to the client...

Copy of the rule and log file excerpt below...

Any ideas???

Cheers

Gavin

 
Source Control:
Where: $HeadURL: http://subversion.card.co.uk/svn/products/Utilities/F5/iRules/++Common++SplunkHTTP.txt $
Last Modified On: $Date: 2012-02-16 12:27:33 +0000 (Thu, 16 Feb 2012) $
Last Modified By: $Author: gavinw $
Revision: $Revision: 57745 $


when RULE_INIT {

set static::LogInvalidRespDebug 1

}

when HTTP_REQUEST priority 100 {

if {$static::LogInvalidRespDebug > 0} { log local0.info "Processin HTTP Request at Priority 100..." }

if { [HTTP::header exists "X-Requested-With"] } {
if {$static::LogInvalidRespDebug > 0} { log local0.info "X-Requested-With HTTP Header Present. Must be AJAX Request." }
return
} else {
if {$static::LogInvalidRespDebug > 0} { log local0.info "Non AJAX Request. Processing further." }

 Only looking for HTML Requests
if { [HTTP::header "Accept"] contains "text/html" } {
if {$static::LogInvalidRespDebug > 0} { log local0.info "Accept Header contains 'text/html'. Recording." }

 Prevent the server from sending a compressed response
 remove the compression offerings from the client
HTTP::header remove "Accept-Encoding"

 Don't allow response data to be chunked
if { [HTTP::version] eq "1.1" } {

 Force downgrade to HTTP 1.0, but still allow keep-alive connections.
 Since HTTP 1.1 is keep-alive by default, and 1.0 isn't,
 we need make sure the headers reflect the keep-alive status.

 Check if this is a keep alive connection
if { [HTTP::header is_keepalive] } {
 Replace the connection header value with "Keep-Alive"
HTTP::header replace "Connection" "Keep-Alive"
}

 Set server side request version to 1.0
 This forces the server to respond without chunking
HTTP::version "1.0"
}

if {$static::LogInvalidRespDebug > 0} { log local0.info "Storing HTTP::request value" }
set req_data [HTTP::request]
if {$static::LogInvalidRespDebug > 0} { log local0.info "\$req_data = $req_data" }


} else {
if {$static::LogInvalidRespDebug > 0} { log local0.info "Non HTML request. Returning. " }
return
}

}

}

when HTTP_RESPONSE {

if {$static::LogInvalidRespDebug > 0} { log local0.info "Processing HTTP Response." }

 Disable the stream filter by default  
STREAM::disable

 Check if response type is text  
if {[HTTP::header value Content-Type] contains "text/html"}{  
if {$static::LogInvalidRespDebug > 0} { log local0.info "HTTP Response Content-Type header contains 'text/html'" }

set lir_stream_find "ACT"
set lir_stream_replace "toe"
set lir_stream_expr "@$lir_stream_find@$lir_stream_replace$lir_stream_find@"
set lir_stream_expr_cmd "STREAM::expression \{\"$lir_stream_expr\"\}"
set lir_stream_enbl_cmd "STREAM::enable"
if { $static::LogInvalidRespDebug > 0 } {
log local0. "\$lir_stream_expr_cmd: $lir_stream_expr_cmd, \$lir_stream_enbl_cmd: $lir_stream_enbl_cmd"
}

 Execute the STREAM::expression command.  Use catch to handle any errors. Save the result to $result
if { [catch {eval $lir_stream_expr_cmd} lir_result] } {
 There was an error trying to set the stream expression.
log local0. "Error setting stream expression ($lir_result). Stream profile not set."
} else {
 No error setting the stream expression, so try to enable the stream filter
 Execute the STREAM::enable command.  Use catch to handle any errors. Save the result to $result
if { [catch {eval $lir_stream_enbl_cmd} result] } {
 There was an error trying to enable the stream filter.
log local0. "Error enabling stream filter ($lir_result): $lir_result"
} else {
if { $static::LogInvalidRespDebug > 0 } {
log local0. "Successfully configured and enabled stream filter"
}
}
}

 Trigger collection for up to 1MB of data
if {[HTTP::header exists "Content-Length"] && [HTTP::header "Content-Length"] <= 1048576}{
set content_length [HTTP::header "Content-Length"]
} else {
set content_length 1048576
}

 Check if $content_length is not set to 0
if { $content_length > 0} {
HTTP::collect $content_length
}

}
}

when HTTP_RESPONSE_DATA {

 do stuff with the payload
set payload [HTTP::payload]
}

when STREAM_MATCHED {

if {$static::LogInvalidRespDebug > 0} { log local0.info "Stream_Matched event triggered" }
log local0. "[IP::client_addr]:[TCP::local_port]: matched: [STREAM::match].]"  
STREAM::replace "[string map {http:// https://} [STREAM::match]]"  
} 

Apr 12 10:23:37 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : Processin HTTP Request at Priority 100...

Apr 12 10:23:37 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : Non AJAX Request. Processing further.

Apr 12 10:23:37 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : Accept Header contains 'text/html'. Recording.

Apr 12 10:23:37 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : Storing HTTP::request value

Apr 12 10:23:37 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : $req_data = GET /CMSBackOffice/home HTTP/1.0 Host: 192.168.151.11 Connection: Keep-Alive Cache-Control: max-age=0 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.152 Safari/535.19 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Referer: http://192.168.151.11/CMSBackOffice/logon Accept-Language: en-GB,en-US;q=0.8,en;q=0.6 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 Cookie: JSESSIONID=5dd69746a6c26778d7f9ef978282; jsonTable={"1148975717": [1334220684671, 1, 0], "1495321059": [1334221943386, 1, 0, 0, 0], "1833417362": [1334220684691, 1, 0, 0, 0], "2105498430": [1334220684687, 1, 0, 0, 0], "2779550008": [1327417348363, 1, 0, 0, 0], "3714451117": [1334220684675, 1, 0, 0, 3], "4138540245": [1334220684697, 1, 0, 0, 0], "v": 31295000}

Apr 12 10:23:38 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : Processing HTTP Response.

Apr 12 10:23:38 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : HTTP Response Content-Type header contains 'text/html'

Apr 12 10:23:38 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : $lir_stream_expr_cmd: STREAM::expression {"@ACT@toeACT@"}, $lir_stream_enbl_cmd: STREAM::enable

Apr 12 10:23:38 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : Successfully configured and enabled stream filter

Hi Gavin,

 

 

Can you make the same request using curl and post the anonymized response headers and content?

 

 

As an aside, you can retrieve the payload that the server includes in the packet(s) TMM needs to collect to parse the response headers using [HTTP::payload] in HTTP_RESPONSE (without calling HTTP::collect). So if the < head > tag is present in the first response packet on successful responses you could potentially log the details for failures in HTTP_RESPONSE without using a stream profile or collecting the payload.

 

 

Aaron

Aaron

As it happens, i've got it working a slightly different way.

Comments welcome on possible improvements though 🙂

Gav

Copy of my rule is:

 

when RULE_INIT {

set static::LogInvalidRespDebug 1

}

when HTTP_REQUEST priority 100 {

if {$static::LogInvalidRespDebug > 0} { log local0.info "Processin HTTP Request at Priority 100..." }

 Bypass by default
set bypass 1

if { [HTTP::header exists "X-Requested-With"] } {
if {$static::LogInvalidRespDebug > 0} { log local0.info "X-Requested-With HTTP Header Present. Must be AJAX Request." }
return
} else {
if {$static::LogInvalidRespDebug > 0} { log local0.info "Non AJAX Request. Processing further." }

 Only looking for HTML Requests
if { [HTTP::header "Accept"] contains "text/html" } {
if {$static::LogInvalidRespDebug > 0} { log local0.info "Accept Header contains 'text/html'. Recording." }

 Prevent the server from sending a compressed response
 remove the compression offerings from the client
HTTP::header remove "Accept-Encoding"

 Don't allow response data to be chunked
if { [HTTP::version] eq "1.1" } {

 Force downgrade to HTTP 1.0, but still allow keep-alive connections.
 Since HTTP 1.1 is keep-alive by default, and 1.0 isn't,
 we need make sure the headers reflect the keep-alive status.

 Check if this is a keep alive connection
if { [HTTP::header is_keepalive] } {
 Replace the connection header value with "Keep-Alive"
HTTP::header replace "Connection" "Keep-Alive"
}

 Set server side request version to 1.0
 This forces the server to respond without chunking
HTTP::version "1.0"
}

if {$static::LogInvalidRespDebug > 0} { log local0.info "Storing HTTP::request value" }
set req_data [HTTP::request]
if {$static::LogInvalidRespDebug > 0} { log local0.info "\$req_data = $req_data" }

 Need to check the repsonse on this one. 
set bypass 0

} else {
if {$static::LogInvalidRespDebug > 0} { log local0.info "Non HTML request. Returning. " }
return
}

}

}

when HTTP_RESPONSE {

if {$static::LogInvalidRespDebug > 0} { log local0.info "Processing HTTP Response." }

if { $bypass } {
if {$static::LogInvalidRespDebug > 0} { log local0.info "Bypassing this response" }
return
} else {

if {$static::LogInvalidRespDebug > 0} { log local0.info "Processing this response" }

 Trigger collection for up to 1MB of data
if {[HTTP::header exists "Content-Length"] && [HTTP::header "Content-Length"] <= 1048576}{
set content_length [HTTP::header "Content-Length"]
} else {
set content_length 1048576
}

 Check if $content_length is not set to 0
if { $content_length > 0} {
HTTP::collect $content_length
}
 Log output. 
set ir_hsl [HSL::open -proto UDP -pool splunk_ir]
HSL::send $ir_hsl "<190>|Checking HTTP Response Data\n"
}
}

when HTTP_RESPONSE_DATA {

if {$static::LogInvalidRespDebug > 0} { log local0.info "Processing HTTP Response data." }

if { $bypass } {
return
} else {

 do stuff with the payload
set payload [HTTP::payload]

if { $payload contains "" } {
if {$static::LogInvalidRespDebug > 0} { log local0. "Payload contains  tag." }
} else { 
log local0.crit "Payload doesnt contain  tag."
HSL::send $ir_hsl "<190>|$clientip requested $uri. Invalid response returned. Request data was: $req_data\n"
}
}

}

 

You could add a check to see if the uncollected payload contains and not bother collecting in that case. Also, you can do a case insensitive comparison to match , , etc:

when RULE_INIT {
set static::LogInvalidRespDebug 1
}

when HTTP_REQUEST priority 100 {
if {$static::LogInvalidRespDebug > 0} { log local0.info "Processin HTTP Request at Priority 100..." }

 Bypass by default
set bypass 1

if { [HTTP::header exists "X-Requested-With"] } {
if {$static::LogInvalidRespDebug > 0} { log local0.info "X-Requested-With HTTP Header Present. Must be AJAX Request." }
return
} else {
if {$static::LogInvalidRespDebug > 0} { log local0.info "Non AJAX Request. Processing further." }

 Only looking for HTML Requests
if { [HTTP::header "Accept"] contains "text/html" } {
if {$static::LogInvalidRespDebug > 0} { log local0.info "Accept Header contains 'text/html'. Recording." }

 Prevent the server from sending a compressed response
 remove the compression offerings from the client
HTTP::header remove "Accept-Encoding"

 Don't allow response data to be chunked
if { [HTTP::version] eq "1.1" } {

 Force downgrade to HTTP 1.0, but still allow keep-alive connections.
 Since HTTP 1.1 is keep-alive by default, and 1.0 isn't,
 we need make sure the headers reflect the keep-alive status.

 Check if this is a keep alive connection
if { [HTTP::header is_keepalive] } {
 Replace the connection header value with "Keep-Alive"
HTTP::header replace "Connection" "Keep-Alive"
}

 Set server side request version to 1.0
 This forces the server to respond without chunking
HTTP::version "1.0"
}

if {$static::LogInvalidRespDebug > 0} { log local0.info "Storing HTTP::request value" }
set req_data [HTTP::request]
if {$static::LogInvalidRespDebug > 0} { log local0.info "\$req_data = $req_data" }

 Need to check the repsonse on this one. 
set bypass 0

} else {
if {$static::LogInvalidRespDebug > 0} { log local0.info "Non HTML request. Returning. " }
return
}

}

}

when HTTP_RESPONSE {

if {$static::LogInvalidRespDebug > 0} { log local0.info "Processing HTTP Response." }

if { $bypass } {
if {$static::LogInvalidRespDebug > 0} { log local0.info "Bypassing this response" }
return
} elseif { [string match -nocase "**" [HTTP::payload]] } {
`if {$static::LogInvalidRespDebug > 0} { log local0. "Uncollected payload contains  tag." }
return
} else {

if {$static::LogInvalidRespDebug > 0} { log local0.info "Processing this response" }

 Trigger collection for up to 1MB of data
if {[HTTP::header exists "Content-Length"] and [HTTP::header "Content-Length"] <= 1048576}{
set content_length [HTTP::header "Content-Length"]
} else {
set content_length 1048576
}

 Check if $content_length is not set to 0
if { $content_length > 0} {
HTTP::collect $content_length
}
 Log output. 
set ir_hsl [HSL::open -proto UDP -pool splunk_ir]
HSL::send $ir_hsl "<190>|Checking HTTP Response Data\n"
}
}

when HTTP_RESPONSE_DATA {

if {$static::LogInvalidRespDebug > 0} { log local0.info "Processing HTTP Response data." }

if { $bypass } {
return
} else {

 do stuff with the payload
set payload [HTTP::payload]

if { [string match -nocase "**" [HTTP::payload]] } {
if {$static::LogInvalidRespDebug > 0} { log local0. "Payload contains  tag." }
} else { 
log local0.crit "Payload doesnt contain  tag."
HSL::send $ir_hsl "<190>|$clientip requested $uri. Invalid response returned. Request data was: $req_data\n"
}
}
}

Aaron

Ah, that looks a bit easier...

 

 

Will give that a go Monday and see how it performs...

 

 

Cheers

 

Gav