HTTP Session Watcher

Problem this snippet solves:

iRule to capture and display HTTP request and response headers (and optionally request payload) for HTTP sessions flowing through the BIG-IP

How to use this snippet:

Screenshots

Code :

### HTTP Session Watcher
### Chad Jenison - c.jenison -at- f5.com 
### version 1.0
### A tool to record and show HTTP Sessions and their underlying requests and response headers (and response payload) on a BIG-IP
### Notes: 
### This rule utilizes a non-trivial amount of table memory and does a pretty good amount of data-plane processing for saving traffic flow data.
### For that reason, you probably want to be careful about deploying it on production virtual servers and begin with your same app/pool on a test virtual server with this iRule enabled
### If you decide to migrate this to a production virtual server, you probably want to take advantage of the options available in RULE_INIT for limiting the traffic the rule captures
### This was tested on BIG-IP 10.2
### The most reliable codepath is to let the rule use its own cookie; use of an external (app) cookie is intended to not impact traffic and reveal use of the tool 
### BIG-IP Generated Redirects are not captured; but do not cause application misbehavior
### Use of the rule on both an HTTP and HTTPS virtual server for the same session has not been tested and likely doesn't work yet. Will fix later.
### Not yet tested with BIG-IP ASM,WA,APM enabled on virtual server
### Textual HTTP Response Status is not displayed; in spite of its presence in the HTTP response. Couldn't find iRule API to access this; only numeric response status
### To-Do: 
### Support combined HTTP&HTTPS access to the same app with session visibility provided in a protocol independent manner
### Clean-Up table display using stylesheets
### Add cr-lf's to raw HTTP request display
### Support HTTP Request Payload Capture
### Escape HTTP Response Payload for display

when RULE_INIT {
# If you want to use existing application session cookie as basis for tracking provide cookie name here
  set static::externalcookiename "JSESSIONID"
# By default this rule uses its own cookie; change this to 1 if using external cookie above
  set static::useexternalcookie 0
# Cookie Name if inserting rule-specific tracking cookie
  set static::trackingcookie "BIGIPHTTPWATCHER"
# If you want to enable IP-based restrictions on session capture, set this static variable to 1
  set static::enablesourceipfilter 0
# NOTE Provide enabled source IP networks in iRule datagroup named "enablednetworks"
# Also, we can capture based on presence of an HTTP request header; set this variable to 1 to enable this restriction
  set static::enablecaptureonheaderpresent 0
  set static::magiccaptureheader "x-bigipsession-capture"
# Also, we can capture based on substring match in User-Agent string; uncomment to enable this restriction
#  set static::useragentmagicstring "bigipsession-capture"
#  set static::useragentmagicstring "Chrome"
# This iRule consumes a lot of processing resources as well as table memory. HTTP response payload is likely the biggest consumer of memory; to disable, set this variable to 1
  set static::disableresponsepayloadcapture 1
# With respect to the memory utilization problem, it also makes sense to set a shorter timeout on the response payload table entries; defaulting to 120 seconds
  set static::responsepayloadtabletimeout 120
## tableentrytimeout in seconds (3600 is one hour)
  set static::tabletimeout 3600
  set static::responsedatalimit 1048576
  if {$static::useexternalcookie} {
    set static::sessioncookie $static::externalcookiename
  } else {
    set static::sessioncookie $static::trackingcookie
  }
  set static::enablemagicurlviewing 1
  set static::magicurlprefix "/f5/bigiphttpwatcher"
  set static::htmlstart "BIG-IP HTTP Session Watcher"
  set static::htmlend ""
}



when CLIENT_ACCEPTED {
  set disablecapture 0
  if {$static::enablesourceipfilter} {
    if {[matchclass [IP::client_addr] equals $::enablednetworks]} {
      #log local0. "Client IP filtering enabled and Client IP matched"
    } else {
      #log local0. "Client IP filtering enabled and Client IP did not match"
      set disablecapture 1
    }
  }
}

# this code implements the session browsing UI and is given a priority of 501 because it doesn't need special early access to the HTTP Request
# performance optimization is not a focus for this code, given that it's just powering the UI
when HTTP_REQUEST priority 501 {
  if {$static::enablemagicurlviewing} {
    if {[HTTP::uri] starts_with $static::magicurlprefix} {
      set magicrequest 1
      log local0. "BIG-IP HTTP Watcher UI Access from [IP::client_addr]"
      set parsedsession [URI::query [HTTP::uri] "session"]
      if {$parsedsession eq ""} {
        set sessionbody {

F5 BIG-IP HTTP Watcher


SessionList


} foreach session [table keys -subtable requestcounter] { set requestcount [table lookup -subtable requestcounter $session] set sessionline "" append sessionbody $sessionline } append sessionbody "
Session IDRequest CountUser Agent (most recent)Source IP(most recent)
" append sessionline $session append sessionline "" append sessionline $requestcount append sessionline "" append sessionline [table lookup -subtable requestheaderUser-Agent $session$last] append sessionline "" append sessionline [table lookup -subtable clientip $session$last] append sessionline "
" set htmlresponse $static::htmlstart$sessionbody$static::htmlend } else { set rawrequest [URI::query [HTTP::uri] "rawrequest"] set rawrequestsent [URI::query [HTTP::uri] "rawrequestsent"] set payload [URI::query [HTTP::uri] "payload"] set parsedrequestnumber [URI::query [HTTP::uri] "requestnumber"] if {$rawrequest eq 1} { set responsebody [table lookup -subtable rawrequest $parsedsession$parsedrequestnumber] set htmlresponse $static::htmlstart$responsebody$static::htmlend } elseif {$payload eq 1} { set htmlresponse [table lookup -subtable responsepayload $parsedsession$parsedrequestnumber] } else { ##we got a session param in the URL, so render requests from the table set requestbody {} ## this for loop will iterate over each request associated with the session, outputting a table row for each request for {set requestnumber 1} {$requestnumber <= [table lookup -subtable requestcounter $parsedsession]} {incr requestnumber} { set requestentry "" append requestbody $requestentry } set htmlresponse $static::htmlstart$requestbody$static::htmlend } } HTTP::respond 200 content $htmlresponse Cache-Control No-Cache Pragma No-Cache } } } when HTTP_REQUEST priority 1 { ## this block of code does all the work of saving HTTP request data to the table structure set magicrequest 0 if {[HTTP::uri] starts_with $static::magicurlprefix} { set magicrequest 1 } if {!$magicrequest} { ## here we're going to decide whether we should cature request to table # we're going to detect a $disablecapture variable and if it's already been set (for example based on the user's client IP not matching allowed networks) don't even bother further evaluation of restrictions if {!$disablecapture} { if {$static::enablecaptureonheaderpresent} { if {[HTTP::header exists $static::magiccaptureheader]} { #log local0. "magic capture header required and is present" } else { #log local0. "magic capture header required and is missing" set disablecapture 1 } } if {[info exists static::useragentmagicstring]} { if {[HTTP::header User-Agent] contains $static::useragentmagicstring} { #log local0. "User-Agent magic string required and present; will capture this request" } else { #log local0. "User-Agent magic string required and not present; disabling capture" set disablecapture 1 } } } if {!$disablecapture} { ##code to do transaction logging is here set nocookie 1 if {[HTTP::cookie exists $static::sessioncookie]} { set nocookie 0 set sessionid [HTTP::cookie $static::sessioncookie] } else { #This code is generating a session cookie value - found on devcentral post by Robert Sutcliffe set rnum [format "%08X" [expr { int(1000000000 * rand() ) } ] ] set cthash [format "%08X" [clock clicks -milliseconds] ] set sessionid "$cthash$rnum" } set requestnumber [table incr -subtable requestcounter -mustexist $sessionid] if {$requestnumber eq ""} { table set -subtable requestcounter $sessionid 1 $static::tabletimeout set requestnumber 1 } #Here is where we store request attributes to table for later viewing table set -subtable rawrequest $sessionid$requestnumber [HTTP::request] $static::tabletimeout table set -subtable clienturi $sessionid$requestnumber [HTTP::uri] $static::tabletimeout table set -subtable clientmethod $sessionid$requestnumber [HTTP::method] $static::tabletimeout table set -subtable clienthttpver $sessionid$requestnumber [HTTP::version] $static::tabletimeout table set -subtable clientrequests $sessionid$requestnumber [HTTP::request] $static::tabletimeout table set -subtable httpreqnum $sessionid$requestnumber [HTTP::request_num] $static::tabletimeout table set -subtable requestheadernames $sessionid$requestnumber [HTTP::header names] $static::tabletimeout foreach header [HTTP::header names] { table set -subtable requestheader$header $sessionid$requestnumber [HTTP::header $header] $static::tabletimeout } table set -subtable clientip $sessionid$requestnumber [IP::client_addr] $static::tabletimeout table set -subtable clientport $sessionid$requestnumber [TCP::client_port] $static::tabletimeout table set -subtable vsip $sessionid$requestnumber [IP::local_addr] $static::tabletimeout table set -subtable vsport $sessionid$requestnumber [TCP::local_port clientside] $static::tabletimeout table set -subtable vsname $sessionid$requestnumber [virtual] $static::tabletimeout table set -subtable cmpunit $sessionid$requestnumber [TMM::cmp_unit] $static::tabletimeout } } } when HTTP_REQUEST_SEND priority 501 { #log local0. "disablecapture: $disablecapture" if {!$disablecapture && !$magicrequest} { clientside { table set -subtable requesturisent $sessionid$requestnumber [HTTP::uri] $static::tabletimeout table set -subtable requestheadernamessent $sessionid$requestnumber [HTTP::header names] $static::tabletimeout table set -subtable requestmethodsent $sessionid$requestnumber [HTTP::method] $static::tabletimeout table set -subtable requesthttpversent $sessionid$requestnumber [HTTP::version] $static::tabletimeout foreach header [HTTP::header names] { table set -subtable requestheadersent$header $sessionid$requestnumber [HTTP::header $header] $static::tabletimeout } } # table set -subtable sentrawrequest $sessionid$requestnumber [HTTP::request] $static::tabletimeout set requestsendtime [clock clicks -milliseconds] } } when HTTP_RESPONSE priority 1 { set dontcaptureresponse 0 set deleterequestentries 0 if {!$disablecapture} { set responsestarttime [clock clicks -milliseconds] # Trigger collection for up to configured limit of response data if {!$static::disableresponsepayloadcapture} { if {[HTTP::header exists "Content-Length"] && [HTTP::header "Content-Length"] <= $static::responsedatalimit}{ set content_length [HTTP::header "Content-Length"] } else { set content_length $static::responsedatalimit } # Check if $content_length has been set and is not set to 0 if { [info exists content_length] && $content_length > 0} { HTTP::collect $content_length } } # we have a serious of if/elseif/else here to deal with the possibilities regarding session cookie usage # *** this scenario below is when the rule itself is going to set a cookie because the request doesn't contain one if {$nocookie && !$static::useexternalcookie} { #code will insert a cookie into 1st response if we are not using external cookie and response didn't contain the cookie HTTP::cookie insert name $static::sessioncookie value $sessionid } elseif {[HTTP::cookie exists $static::sessioncookie]} { # *** this scenario below is used when we detect the server setting an external session cookie #we're hoping to find the server issuing a set-cookie right here for the external session cookie being used if {$requestnumber eq 1} { # We're here because this is a first hit and we are seeing the server do a set-cookie set deleterequestentries 1 } else { # We're here because the server is doing a set-cookie for the session cookie and it's not the first request # we want to avoid destroying the old table entries # and let's populate the old session's final request with the response we see here (it'll also be added to the new session's first request entry) table set -subtable responsestatus $sessionid$requestnumber [HTTP::status] $static::tabletimeout table set -subtable responseheadernames $sessionid$requestnumber [HTTP::header names] $static::tabletimeout table set -subtable serverip $sessionid$requestnumber [IP::server_addr] $static::tabletimeout table set -subtable serverport $sessionid$requestnumber [TCP::server_port] $static::tabletimeout table set -subtable serversideip $sessionid$requestnumber [IP::local_addr] $static::tabletimeout table set -subtable serversideport $sessionid$requestnumber [TCP::local_port] $static::tabletimeout table set -subtable httprespnum $sessionid$requestnumber [HTTP::request_num] $static::tabletimeout table set -subtable responsestartdelay $sessionid$requestnumber [expr {$responsestarttime - $requestsendtime}] $static::tabletimeout foreach header [HTTP::header names] { table set -subtable responseheader$header $sessionid$requestnumber [HTTP::header $header] $static::tabletimeout } } set newrequestnumber 1 set priorsessionid $sessionid set sessionid [HTTP::cookie $static::sessioncookie] table set -subtable requestcounter $sessionid 1 $static::tabletimeout # this code will migrate request table entries to use the new external session cookie value as key and then delete the table entries using the (temp) priorsessionid as key table set -subtable rawrequest $sessionid$newrequestnumber [table lookup -subtable rawrequest $priorsessionid$requestnumber] $static::tabletimeout table set -subtable clienturi $sessionid$newrequestnumber [table lookup -subtable clienturi $priorsessionid$requestnumber] $static::tabletimeout table set -subtable clientmethod $sessionid$newrequestnumber [table lookup -subtable clientmethod $priorsessionid$requestnumber] $static::tabletimeout table set -subtable clienthttpver $sessionid$newrequestnumber [table lookup -subtable clienthttpver $priorsessionid$requestnumber] $static::tabletimeout table set -subtable clientrequests $sessionid$newrequestnumber [table lookup -subtable clientrequests $priorsessionid$requestnumber] $static::tabletimeout table set -subtable httpreqnum $sessionid$newrequestnumber [table lookup -subtable httpreqnumber $priorsessionid$requestnumber] $static::tabletimeout table set -subtable requestheadernames $sessionid$newrequestnumber [table lookup -subtable requestheadernames $priorsessionid$requestnumber] $static::tabletimeout foreach header [table lookup -subtable requestheadernames $priorsessionid$requestnumber] { table set -subtable requestheader$header $sessionid$newrequestnumber [table lookup -subtable requestheader$header $priorsessionid$requestnumber] $static::tabletimeout } table set -subtable clientip $sessionid$newrequestnumber [table lookup -subtable clientip $priorsessionid$requestnumber] $static::tabletimeout table set -subtable clientport $sessionid$newrequestnumber [table lookup -subtable clientport $priorsessionid$requestnumber] $static::tabletimeout table set -subtable vsip $sessionid$newrequestnumber [table lookup -subtable vsip $priorsessionid$requestnumber] $static::tabletimeout table set -subtable vsport $sessionid$newrequestnumber [table lookup -subtable vsport $priorsessionid$requestnumber] $static::tabletimeout table set -subtable vsname $sessionid$newrequestnumber [table lookup -subtable vsname $priorsessionid$requestnumber] $static::tabletimeout table set -subtable cmpunit $sessionid$newrequestnumber [table lookup -subtable cmpunit $priorsessionid$requestnumber] $static::tabletimeout table set -subtable requesturisent $sessionid$newrequestnumber [table lookup -subtable requesturisent $priorsessionid$requestnumber] $static::tabletimeout table set -subtable requestheadernamessent $sessionid$newrequestnumber [table lookup -subtable requestheadernamessent $priorsessionid$requestnumber] $static::tabletimeout table set -subtable requestmethodsent $sessionid$newrequestnumber [table lookup -subtable requestmethodsent $priorsessionid$requestnumber] $static::tabletimeout table set -subtable requesthttpversent $sessionid$newrequestnumber [table lookup -subtable requesthttpversent $priorsessionid$requestnumber] $static::tabletimeout foreach header [table lookup -subtable requestheadernamessent $priorsessionid$requestnumber] { table set -subtable requestheadersent$header $sessionid$newrequestnumber [table lookup -subtable requestheadersent$header $priorsessionid$requestnumber] $static::tabletimeout } set requestnumber 1 } elseif {$nocookie} { # lacking cookie in request and cookie not being set in response; we'll avoid capturing response and delete the initial request table entries set dontcaptureresponse 1 set deleterequestentries 1 set priorsessionid $sessionid } if {!$dontcaptureresponse} { table set -subtable responsestatus $sessionid$requestnumber [HTTP::status] $static::tabletimeout table set -subtable responseheadernames $sessionid$requestnumber [HTTP::header names] $static::tabletimeout table set -subtable serverip $sessionid$requestnumber [IP::server_addr] $static::tabletimeout table set -subtable serverport $sessionid$requestnumber [TCP::server_port] $static::tabletimeout table set -subtable serversideip $sessionid$requestnumber [IP::local_addr] $static::tabletimeout table set -subtable serversideport $sessionid$requestnumber [TCP::local_port] $static::tabletimeout table set -subtable httprespnum $sessionid$requestnumber [HTTP::request_num] $static::tabletimeout table set -subtable responsestartdelay $sessionid$requestnumber [expr {$responsestarttime - $requestsendtime}] $static::tabletimeout foreach header [HTTP::header names] { table set -subtable responseheader$header $sessionid$requestnumber [HTTP::header $header] $static::tabletimeout } } if {$deleterequestentries} { # we're deleting HTTP request table entries (and presumably the value of requestnumber is 1) table delete -subtable rawrequest $priorsessionid$requestnumber table delete -subtable clienturi $priorsessionid$requestnumber table delete -subtable clientmethod $priorsessionid$requestnumber table delete -subtable clienthttpver $priorsessionid$requestnumber table delete -subtable clientrequests $priorsessionid$requestnumber table delete -subtable httpreqnum $priorsessionid$requestnumber foreach header [table lookup -subtable requestheadernames $priorsessionid$requestnumber] { table delete -subtable requestheader$header $priorsessionid$requestnumber } table delete -subtable requestheadernames $priorsessionid$requestnumber table delete -subtable clientip $priorsessionid$requestnumber table delete -subtable clientport $priorsessionid$requestnumber table delete -subtable vsip $priorsessionid$requestnumber table delete -subtable vsport $priorsessionid$requestnumber table delete -subtable vsname $priorsessionid$requestnumber table delete -subtable cmpunit $priorsessionid$requestnumber table delete -subtable requesturisent $priorsessionid$requestnumber table delete -subtable requestheadernamessent $priorsessionid$requestnumber table delete -subtable requestmethodsent $priorsessionid$requestnumber table delete -subtable requesthttpversent $priorsessionid$requestnumber foreach header [HTTP::header names] { table delete -subtable requestheadersent$header $priorsessionid$requestnumber } table delete -subtable requestcounter $priorsessionid } } } when HTTP_RESPONSE_DATA { if {!$disablecapture} { #record time set responsercvtime [clock clicks -milliseconds] #store payload to table table set -subtable responsepayload $sessionid$requestnumber [HTTP::payload] table set -subtable responsercvdelay $sessionid$requestnumber [expr {$responsercvtime - $requestsendtime}] $static::responsepayloadtabletimeout } }
Request NumberBIG-IP InfoHTTP Request(received from client)HTTP Request(sent to server)HTTP Response
" append requestentry "$requestnumber" append requestentry "Raw Request Rcvd" set responsepayload [table lookup -subtable responsepayload $parsedsession$requestnumber] if {$responsepayload eq ""} { } else { append requestentry "Response Payload" } append requestentry "" append requestentry "Client IP & Port: " append requestentry [table lookup -subtable clientip $parsedsession$requestnumber] append requestentry ":" append requestentry [table lookup -subtable clientport $parsedsession$requestnumber] append requestentry "" append requestentry "Client IP & Port(Server-side): " append requestentry [table lookup -subtable serversideip $parsedsession$requestnumber] append requestentry ":" append requestentry [table lookup -subtable serversideport $parsedsession$requestnumber] append requestentry "" append requestentry "Virtual Server IP & Port: " append requestentry [table lookup -subtable vsip $parsedsession$requestnumber] append requestentry ":" append requestentry [table lookup -subtable vsport $parsedsession$requestnumber] append requestentry "" append requestentry "Server IP & Port: " append requestentry [table lookup -subtable serverip $parsedsession$requestnumber] append requestentry ":" append requestentry [table lookup -subtable serverport $parsedsession$requestnumber] append requestentry "Virtual Name: " append requestentry [table lookup -subtable vsname $parsedsession$requestnumber] append requestentry "CMP Unit: " append requestentry [table lookup -subtable cmpunit $parsedsession$requestnumber] append requestentry "HTTP Req Num on Socket: " append requestentry [table lookup -subtable httpreqnum $parsedsession$requestnumber] append requestentry "HTTP Resp Num on Socket: " append requestentry [table lookup -subtable httprespnum $parsedsession$requestnumber] append requestentry "Client IP Continent: " append requestentry [whereis [table lookup -subtable clientip $parsedsession$requestnumber] continent] append requestentry "Client IP Country: " append requestentry [whereis [table lookup -subtable clientip $parsedsession$requestnumber] country] append requestentry "Client IP State: " append requestentry [whereis [table lookup -subtable clientip $parsedsession$requestnumber] state] append requestentry "Response Start Delay: " append requestentry [table lookup -subtable responsestartdelay $parsedsession$requestnumber] if {$responsepayload eq ""} { } else { append requestentry "Response Finish Delay: " append requestentry [table lookup -subtable responsercvdelay $parsedsession$requestnumber] } append requestentry "" ##Output HTTP request information here append requestentry [table lookup -subtable clientmethod $parsedsession$requestnumber] append requestentry " " append requestentry [table lookup -subtable clienturi $parsedsession$requestnumber] append requestentry " HTTP/" append requestentry [table lookup -subtable clienthttpver $parsedsession$requestnumber] append requestentry "" foreach header [table lookup -subtable requestheadernames $parsedsession$requestnumber] { append requestentry $header append requestentry ": " append requestentry [table lookup -subtable requestheader$header $parsedsession$requestnumber] append requestentry "" } append requestentry "" ##Output Request Sent to Server (post-modification) here append requestentry [table lookup -subtable requestmethodsent $parsedsession$requestnumber] append requestentry " " append requestentry [table lookup -subtable requesturisent $parsedsession$requestnumber] append requestentry " HTTP/" append requestentry [table lookup -subtable requesthttpversent $parsedsession$requestnumber] append requestentry "" foreach header [table lookup -subtable requestheadernamessent $parsedsession$requestnumber] { append requestentry $header append requestentry ": " append requestentry [table lookup -subtable requestheadersent$header $parsedsession$requestnumber] append requestentry "" } append requestentry "" ##Response Output Code append requestentry "Response Status:" append requestentry [table lookup -subtable responsestatus $parsedsession$requestnumber] append requestentry "" append requestentry "Response Headers:" foreach header [table lookup -subtable responseheadernames $parsedsession$requestnumber] { append requestentry $header append requestentry ": " append requestentry [table lookup -subtable responseheader$header $parsedsession$requestnumber] append requestentry "" } append requestentry "
Published Mar 18, 2015
Version 1.0

Was this article helpful?

No CommentsBe the first to comment