Forum Discussion

GavinW_29074's avatar
GavinW_29074
Icon for Nimbostratus rankNimbostratus
Apr 12, 2012

Check HTTP Response Data for string

Hi there,

 

 

We've got an intermittent issue with one of our applications which I'm trying to get to understand a bit better using our F5's and iRules...

 

 

Basically, sometimes the application fails to include the head block in the HTML responses.

 

 

I want to be able to use an iRule to log every time this happens?

 

 

What I'm trying to get is the most economical and reliable way of searching the response data for the given string.

 

Is this something that can be achieved using STREAM profiles?

 

 

Comments welcome...

 

 

Cheers

 

Gav

 

  • Hi there,

    So I went ahead and am trying to use Stream Expressions to check for the string...

    I've got the iRule compiling the expression and it appears to be setting it ok. However it's making NO impact on the resultant HTML returned to the client...

    Copy of the rule and log file excerpt below...

    Any ideas???

    Cheers

    Gavin

     
    Source Control:
    Where: $HeadURL: http://subversion.card.co.uk/svn/products/Utilities/F5/iRules/++Common++SplunkHTTP.txt $
    Last Modified On: $Date: 2012-02-16 12:27:33 +0000 (Thu, 16 Feb 2012) $
    Last Modified By: $Author: gavinw $
    Revision: $Revision: 57745 $
    
    
    when RULE_INIT {
    
    set static::LogInvalidRespDebug 1
    
    }
    
    when HTTP_REQUEST priority 100 {
    
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Processin HTTP Request at Priority 100..." }
    
    if { [HTTP::header exists "X-Requested-With"] } {
    if {$static::LogInvalidRespDebug > 0} { log local0.info "X-Requested-With HTTP Header Present. Must be AJAX Request." }
    return
    } else {
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Non AJAX Request. Processing further." }
    
     Only looking for HTML Requests
    if { [HTTP::header "Accept"] contains "text/html" } {
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Accept Header contains 'text/html'. Recording." }
    
     Prevent the server from sending a compressed response
     remove the compression offerings from the client
    HTTP::header remove "Accept-Encoding"
    
     Don't allow response data to be chunked
    if { [HTTP::version] eq "1.1" } {
    
     Force downgrade to HTTP 1.0, but still allow keep-alive connections.
     Since HTTP 1.1 is keep-alive by default, and 1.0 isn't,
     we need make sure the headers reflect the keep-alive status.
    
     Check if this is a keep alive connection
    if { [HTTP::header is_keepalive] } {
     Replace the connection header value with "Keep-Alive"
    HTTP::header replace "Connection" "Keep-Alive"
    }
    
     Set server side request version to 1.0
     This forces the server to respond without chunking
    HTTP::version "1.0"
    }
    
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Storing HTTP::request value" }
    set req_data [HTTP::request]
    if {$static::LogInvalidRespDebug > 0} { log local0.info "\$req_data = $req_data" }
    
    
    } else {
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Non HTML request. Returning. " }
    return
    }
    
    }
    
    }
    
    when HTTP_RESPONSE {
    
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Processing HTTP Response." }
    
     Disable the stream filter by default  
    STREAM::disable
    
     Check if response type is text  
    if {[HTTP::header value Content-Type] contains "text/html"}{  
    if {$static::LogInvalidRespDebug > 0} { log local0.info "HTTP Response Content-Type header contains 'text/html'" }
    
    set lir_stream_find "ACT"
    set lir_stream_replace "toe"
    set lir_stream_expr "@$lir_stream_find@$lir_stream_replace$lir_stream_find@"
    set lir_stream_expr_cmd "STREAM::expression \{\"$lir_stream_expr\"\}"
    set lir_stream_enbl_cmd "STREAM::enable"
    if { $static::LogInvalidRespDebug > 0 } {
    log local0. "\$lir_stream_expr_cmd: $lir_stream_expr_cmd, \$lir_stream_enbl_cmd: $lir_stream_enbl_cmd"
    }
    
     Execute the STREAM::expression command.  Use catch to handle any errors. Save the result to $result
    if { [catch {eval $lir_stream_expr_cmd} lir_result] } {
     There was an error trying to set the stream expression.
    log local0. "Error setting stream expression ($lir_result). Stream profile not set."
    } else {
     No error setting the stream expression, so try to enable the stream filter
     Execute the STREAM::enable command.  Use catch to handle any errors. Save the result to $result
    if { [catch {eval $lir_stream_enbl_cmd} result] } {
     There was an error trying to enable the stream filter.
    log local0. "Error enabling stream filter ($lir_result): $lir_result"
    } else {
    if { $static::LogInvalidRespDebug > 0 } {
    log local0. "Successfully configured and enabled stream filter"
    }
    }
    }
    
     Trigger collection for up to 1MB of data
    if {[HTTP::header exists "Content-Length"] && [HTTP::header "Content-Length"] <= 1048576}{
    set content_length [HTTP::header "Content-Length"]
    } else {
    set content_length 1048576
    }
    
     Check if $content_length is not set to 0
    if { $content_length > 0} {
    HTTP::collect $content_length
    }
    
    }
    }
    
    when HTTP_RESPONSE_DATA {
    
     do stuff with the payload
    set payload [HTTP::payload]
    }
    
    when STREAM_MATCHED {
    
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Stream_Matched event triggered" }
    log local0. "[IP::client_addr]:[TCP::local_port]: matched: [STREAM::match].]"  
    STREAM::replace "[string map {http:// https://} [STREAM::match]]"  
    } 

    Apr 12 10:23:37 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : Processin HTTP Request at Priority 100...

    Apr 12 10:23:37 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : Non AJAX Request. Processing further.

    Apr 12 10:23:37 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : Accept Header contains 'text/html'. Recording.

    Apr 12 10:23:37 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : Storing HTTP::request value

    Apr 12 10:23:37 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : $req_data = GET /CMSBackOffice/home HTTP/1.0 Host: 192.168.151.11 Connection: Keep-Alive Cache-Control: max-age=0 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.152 Safari/535.19 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Referer: http://192.168.151.11/CMSBackOffice/logon Accept-Language: en-GB,en-US;q=0.8,en;q=0.6 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 Cookie: JSESSIONID=5dd69746a6c26778d7f9ef978282; jsonTable={"1148975717": [1334220684671, 1, 0], "1495321059": [1334221943386, 1, 0, 0, 0], "1833417362": [1334220684691, 1, 0, 0, 0], "2105498430": [1334220684687, 1, 0, 0, 0], "2779550008": [1327417348363, 1, 0, 0, 0], "3714451117": [1334220684675, 1, 0, 0, 3], "4138540245": [1334220684697, 1, 0, 0, 0], "v": 31295000}

    Apr 12 10:23:38 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : Processing HTTP Response.

    Apr 12 10:23:38 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : HTTP Response Content-Type header contains 'text/html'

    Apr 12 10:23:38 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : $lir_stream_expr_cmd: STREAM::expression {"@ACT@toeACT@"}, $lir_stream_enbl_cmd: STREAM::enable

    Apr 12 10:23:38 tmm info tmm[9166]: Rule /Common/LogInvalidResponses : Successfully configured and enabled stream filter
  • Hi Gavin,

     

     

    Can you make the same request using curl and post the anonymized response headers and content?

     

     

    As an aside, you can retrieve the payload that the server includes in the packet(s) TMM needs to collect to parse the response headers using [HTTP::payload] in HTTP_RESPONSE (without calling HTTP::collect). So if the < head > tag is present in the first response packet on successful responses you could potentially log the details for failures in HTTP_RESPONSE without using a stream profile or collecting the payload.

     

     

    Aaron
  • Aaron

    As it happens, i've got it working a slightly different way.

    Comments welcome on possible improvements though 🙂

    Gav

    Copy of my rule is:

     
    
    when RULE_INIT {
    
    set static::LogInvalidRespDebug 1
    
    }
    
    when HTTP_REQUEST priority 100 {
    
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Processin HTTP Request at Priority 100..." }
    
     Bypass by default
    set bypass 1
    
    if { [HTTP::header exists "X-Requested-With"] } {
    if {$static::LogInvalidRespDebug > 0} { log local0.info "X-Requested-With HTTP Header Present. Must be AJAX Request." }
    return
    } else {
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Non AJAX Request. Processing further." }
    
     Only looking for HTML Requests
    if { [HTTP::header "Accept"] contains "text/html" } {
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Accept Header contains 'text/html'. Recording." }
    
     Prevent the server from sending a compressed response
     remove the compression offerings from the client
    HTTP::header remove "Accept-Encoding"
    
     Don't allow response data to be chunked
    if { [HTTP::version] eq "1.1" } {
    
     Force downgrade to HTTP 1.0, but still allow keep-alive connections.
     Since HTTP 1.1 is keep-alive by default, and 1.0 isn't,
     we need make sure the headers reflect the keep-alive status.
    
     Check if this is a keep alive connection
    if { [HTTP::header is_keepalive] } {
     Replace the connection header value with "Keep-Alive"
    HTTP::header replace "Connection" "Keep-Alive"
    }
    
     Set server side request version to 1.0
     This forces the server to respond without chunking
    HTTP::version "1.0"
    }
    
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Storing HTTP::request value" }
    set req_data [HTTP::request]
    if {$static::LogInvalidRespDebug > 0} { log local0.info "\$req_data = $req_data" }
    
     Need to check the repsonse on this one. 
    set bypass 0
    
    } else {
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Non HTML request. Returning. " }
    return
    }
    
    }
    
    }
    
    when HTTP_RESPONSE {
    
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Processing HTTP Response." }
    
    if { $bypass } {
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Bypassing this response" }
    return
    } else {
    
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Processing this response" }
    
     Trigger collection for up to 1MB of data
    if {[HTTP::header exists "Content-Length"] && [HTTP::header "Content-Length"] <= 1048576}{
    set content_length [HTTP::header "Content-Length"]
    } else {
    set content_length 1048576
    }
    
     Check if $content_length is not set to 0
    if { $content_length > 0} {
    HTTP::collect $content_length
    }
     Log output. 
    set ir_hsl [HSL::open -proto UDP -pool splunk_ir]
    HSL::send $ir_hsl "<190>|Checking HTTP Response Data\n"
    }
    }
    
    when HTTP_RESPONSE_DATA {
    
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Processing HTTP Response data." }
    
    if { $bypass } {
    return
    } else {
    
     do stuff with the payload
    set payload [HTTP::payload]
    
    if { $payload contains "" } {
    if {$static::LogInvalidRespDebug > 0} { log local0. "Payload contains  tag." }
    } else { 
    log local0.crit "Payload doesnt contain  tag."
    HSL::send $ir_hsl "<190>|$clientip requested $uri. Invalid response returned. Request data was: $req_data\n"
    }
    }
    
    }
    
     
  • You could add a check to see if the uncollected payload contains and not bother collecting in that case. Also, you can do a case insensitive comparison to match , , etc:

    
    when RULE_INIT {
    set static::LogInvalidRespDebug 1
    }
    
    when HTTP_REQUEST priority 100 {
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Processin HTTP Request at Priority 100..." }
    
     Bypass by default
    set bypass 1
    
    if { [HTTP::header exists "X-Requested-With"] } {
    if {$static::LogInvalidRespDebug > 0} { log local0.info "X-Requested-With HTTP Header Present. Must be AJAX Request." }
    return
    } else {
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Non AJAX Request. Processing further." }
    
     Only looking for HTML Requests
    if { [HTTP::header "Accept"] contains "text/html" } {
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Accept Header contains 'text/html'. Recording." }
    
     Prevent the server from sending a compressed response
     remove the compression offerings from the client
    HTTP::header remove "Accept-Encoding"
    
     Don't allow response data to be chunked
    if { [HTTP::version] eq "1.1" } {
    
     Force downgrade to HTTP 1.0, but still allow keep-alive connections.
     Since HTTP 1.1 is keep-alive by default, and 1.0 isn't,
     we need make sure the headers reflect the keep-alive status.
    
     Check if this is a keep alive connection
    if { [HTTP::header is_keepalive] } {
     Replace the connection header value with "Keep-Alive"
    HTTP::header replace "Connection" "Keep-Alive"
    }
    
     Set server side request version to 1.0
     This forces the server to respond without chunking
    HTTP::version "1.0"
    }
    
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Storing HTTP::request value" }
    set req_data [HTTP::request]
    if {$static::LogInvalidRespDebug > 0} { log local0.info "\$req_data = $req_data" }
    
     Need to check the repsonse on this one. 
    set bypass 0
    
    } else {
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Non HTML request. Returning. " }
    return
    }
    
    }
    
    }
    
    when HTTP_RESPONSE {
    
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Processing HTTP Response." }
    
    if { $bypass } {
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Bypassing this response" }
    return
    } elseif { [string match -nocase "**" [HTTP::payload]] } {
    `if {$static::LogInvalidRespDebug > 0} { log local0. "Uncollected payload contains  tag." }
    return
    } else {
    
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Processing this response" }
    
     Trigger collection for up to 1MB of data
    if {[HTTP::header exists "Content-Length"] and [HTTP::header "Content-Length"] <= 1048576}{
    set content_length [HTTP::header "Content-Length"]
    } else {
    set content_length 1048576
    }
    
     Check if $content_length is not set to 0
    if { $content_length > 0} {
    HTTP::collect $content_length
    }
     Log output. 
    set ir_hsl [HSL::open -proto UDP -pool splunk_ir]
    HSL::send $ir_hsl "<190>|Checking HTTP Response Data\n"
    }
    }
    
    when HTTP_RESPONSE_DATA {
    
    if {$static::LogInvalidRespDebug > 0} { log local0.info "Processing HTTP Response data." }
    
    if { $bypass } {
    return
    } else {
    
     do stuff with the payload
    set payload [HTTP::payload]
    
    if { [string match -nocase "**" [HTTP::payload]] } {
    if {$static::LogInvalidRespDebug > 0} { log local0. "Payload contains  tag." }
    } else { 
    log local0.crit "Payload doesnt contain  tag."
    HSL::send $ir_hsl "<190>|$clientip requested $uri. Invalid response returned. Request data was: $req_data\n"
    }
    }
    }
    

    Aaron
  • Ah, that looks a bit easier...

     

     

    Will give that a go Monday and see how it performs...

     

     

    Cheers

     

    Gav