Referral Tracking With iRules

Introduction

We’ve been having a lot of fun with tables and the Google Charts API. Colin kicked it off with his Heatmaps series and since then we’ve come up with a handful of other implementations. I had recently been doing some SEO (Search Engine Optimization) work on one of my other projects and wanted to see who was sending the traffic my way. This can be done by grepping through server access logs, compiling the data, and generating charts, but that seemed like too much effort. Instead, I decided to write an iRule that would do all of this for me. Adding an iRule to my LTM seemed a much easier solution than modifying or even accessing the multiple origin servers.

The Nuts And Bolts

When an HTTP request arrives at our virtual we inspect the “Referer” [sic] header. If it is a new referrer, we add the referrer to the ‘referrers’ table as the key, set the value to 1, and the timeout and lifetime values to indefinite. If it is a repeat referrer, we increment the number of referrals by 1. This process repeats with each additional request and after some period of time we should have some meaningful data. You may add the follow test code at a higher priority to emulate this behavior:

   1: rule referral_tracker_test {

   2:     priority 10

3:

   4:     when HTTP_REQUEST {

   5:         set test_referrers [list "www.google.com" "bing.com" "search.yahoo.com" "blogs.example.com" "none"]

   6:         set rand_test_referrer [lindex $test_referrers [expr {int(rand()*[llength $test_referrers])}]]

   7:         log local0. "The randomly selected referrer was $rand_test_referrer"

   8:         HTTP::header replace "Referer" "http://$rand_test_referrer"

   9:         HTTP::header replace "Host" "www.example.com"

  10:     }

  11: }

Once the iRule has been in place long enough to collect a number of requests, the data may be viewed by accessing http://<virtual_address>/admin?action=report_referrers. When a request is received for the administrator interface, the user will first be asked for their basic access authentication (see HTTP Basic Access Authentication iRule Style) credentials for the virtual. Once they have been granted access, the chart will then be assembled by looping through each of the referrers and appending the chd (chart data) and chl (chart label) parameters of the Google Charts API URL. Once this is complete, a chart will then be returned to the user showing the proportion of requests from different referrers.

If the data needs to be reset at any time, the “Reset referrer counters” link can be clicked and the referrers will be looped through once again and each key will be deleted. When it all comes together, this is the resulting iRule:

   1: when RULE_INIT {

   2:     set static::response_header "<html><head><title>Top referrers</title></head><body>"

   3:     set static::response_footer "</body></html>"

   4: }

5:

   6: when HTTP_REQUEST {

   7:     set referrer_host [URI::host [HTTP::header value Referer]]

8:

   9:     if { $referrer_host eq "" } {

  10:         set referrer_host "none"

  11:     }

12:

  13:     if { [table incr -subtable referrers -mustexist $referrer_host] eq "" } {

  14:         table set -subtable referrers $referrer_host 1 indefinite indefinite

  15:     } else {

  16:         table incr -subtable referrers $referrer_host

  17:     }

18:

  19:     if { [HTTP::path] eq "/admin" } {

  20:         binary scan [md5 [HTTP::password]] H* password

21:

  22:         if { [class lookup [HTTP::username] $::authorized_users] equals $password } {

  23:             log local0. "User [HTTP::username] has been authorized to access virtual server [virtual name]"

24:

  25:             switch -glob [URI::query [HTTP::uri] action] {

  26:                 reset_referrers {

  27:                     foreach referrer [table keys -subtable referrers] {

  28:                         table delete -subtable referrers $referrer

  29:                     }

30:

  31:                     HTTP::respond 200 content "$static::response_header <center><br><h2>Referrer counters \

  32:                         reset</h2><br><a href=\"/admin?action=report_referrers\">Back to referrer chart</a> \

  33:                         </center>$static::response_footer"

  34:                 }

  35:                 report_referrers -

  36:                 default {

  37:                     set chart "http://chart.apis.google.com/chart?cht=pc&chs=550x300&chts=003399,20&"

  38:                     append chart "chco=003399,CCFFFF&chtt=Top%20referrers%20for%20[HTTP::host]"

39:

  40:                     set chd ""

  41:                     set chl ""

42:

  43:                     foreach referrer [table keys -subtable referrers] {

  44:                         append chd "," [table lookup -subtable referrers $referrer]

  45:                         append chl "|" [URI::encode "$referrer ([table lookup -subtable referrers $referrer] hits)"]

  46:                     }

47:

  48:                     append chart "&" "chd=t:[string trimleft $chd ,]"

  49:                     append chart "&" "chl=[string trimleft $chl |]"

50:

  51:                     HTTP::respond 200 content "$static::response_header <center><br><img src=\"$chart\" /><br> \

  52:                         <br><a href=\"/admin?action=reset_referrers\">Reset referrer counters</a></center> \

  53:                         $static::response_footer"

  54:                 }

  55:             }

  56:         } else {

  57:             if { [string length [HTTP::password]] != 0 } {

  58:                 log local0. "User [HTTP::username] has been denied access to virtual server [virtual name]"

  59:             }

60:

  61:             HTTP::respond 401 WWW-Authenticate "Basic realm=\"Secured Area\""

  62:         }

63:

  64:     }

  65: }

*Note: Don’t forget to add the “authorized_users” data group. The iRule will error and send a TCP reset for the virtual if it is absent.

Conclusion

Whether you’re tracking search engine referrals, decommissioning an old link, or identifying a media spike, this iRule could come in handy. We will have more features plus more iRule and Google Charts kung fu in the near future. In the meantime, we hope this little iRule will help you in profiling referrers in your environment.

Published Oct 11, 2010

Version 1.0