First Client Request

I've got a couple of suggestions. One is to do it based on the URI... something like this:

 

 

when HTTP_REQUEST {

 

if { [HTTP::uri] ends_with ".asp" } {

 

log local0. [HTTP::uri]

 

}

 

}

 

 

If all of your main pages and only your main pages end in .asp then that will work. You could even code a few exceptions in there.

 

 

A second option is to look at the content type of the responses... for example:

 

 

when HTTP_RESPONSE {

 

if { [HTTP::header "Content-type"] eq "text/html"} {

 

log local0. [HTTP::uri]

 

}

 

}

 

 

This assumes that your server properly sets the content type of *.css, *.js, images, etc, to something other than text/html.

 

 

A third method would be to use the "Referer" (sic) header. The browser should include this header in every request. The plus side of this method is that it is not dependent on file extension and/or content-types. The down-side is this would be a bit more complicated. The simple method is to just log every Referer header like this:

 

 

when HTTP_REQUEST {

 

log local0. [HTTP::header "Referer"]

 

}

 

 

Here are the problems with this:

 

 

1) The most obvious problem will be that if a user visits a page that references a number of images, stylesheets, etc, you will get the same Referer URI in several requests. The easy solution is to post-process the logs with a simple shell script or Perl script that will consolidate repeated URIs that are the exactly the same from the same client down to a single entry. The hard part there is identifying the client uniquely -- remote IP is OK but could cause you to over-consolidate for things like AOL users behind a common proxy, etc. Ideally you'd log a session variable or username, etc, to be used in the log elimination. Another solution is to have the iRule remember the last Referer logged for each unique session and to not log that same Referer again -- basically do the consolidation in the iRule. The concern here is performance and memory utilization -- I would use the UIE persistence table to log the last Referer logged for each session so it is automatically flushed out, etc.

 

2) Another problem is that when the user loads the first page on your site their Referer will be from the last site (such as the link to their Google search results page). You could fix this by only logging the Referer if it matches your domain name, etc.

 

3) Another problem is that users can forge their Referer header (important consideration if users might be motivated to forge their browsing of your site)

 

4) A final problem is that if any page does not refer to any objects such as stylesheets, images, javascript, etc, and that is the last page the user visits, then you will never get a request with that page in the Referer and it won't be logged. This matters because even if a page refers to these objects if they are already cached such that not even a conditional get is required then you won't get any requests with that page as the Referer.