Heatmaps, iRules Style: Part2

Last week I talked about generating a heat map 100% via iRules, thanks to the geolocation magic in LTM systems, and the good people over at Google letting us use their charting API. This was an outstanding way to visualize the the traffic coming to your application. For those interested in metrics it provides a great way to see this data in a visually pleasing manner. That said, it was pretty basic. All it showed was the United States which, for anyone that has used the internet much, is obviously not representative of the entire web. To be truly useful we’ll need to show the entire world. That’s simple enough. We’ll update the region the map we’re drawing zooms to, so it will look more like this:

Let’s take a look at how this is going to work. Since we were collecting data based on state abbreviations before, we’ll need to first switch that up to use country codes instead. We’ll then change up our Google call so that we’re setting the range covered by the map to the entire world, rather than just the US. While we’re at it, let’s change the name of the subtable we’re using from states to countries, just to keep things more clear. What we end up with is some code that looks very familiar, if you’ve already seen last week’s solution, with a few minor changes:

set chld ""
 set chd ""
 foreach country [table keys -subtable countries] {
 append chld $country
 append chd "[table lookup -subtable countries $country],"
 }
 set chd [string trimright $chd ","]
 HTTP::respond 200 content "<HTML><center>Here is your site's usage by Country: <img src='http://chart.apis.google.com/chart?cht=t&chd=&chs=440x220&chtm=world&chd=t:$chd&chld=$chld&chco=f5f5f5,edf0d4,6c9642,365e24,13390a' border='0'> br> <a href='/resetmap'>Reset All Counters</a></center></HTML>"

So using that in place of the similar logic in last week’s solution you can get a simple world view of the traffic passing through your site. That’s great and all, but what if you can’t see the detail you’re looking for? What if you want to see the details of Asia’s traffic and be able to decipher the patterns in Japan and the Middle East? What we really need is to build a simple interface to make this more of an application, and less of a single image displayed on a web page.

Well, first of all, we already have all the data collected that we’ll need, if you think about it. We’re already tracking the requests per country, so all we need to do is build out options to allow for users to click a link and zoom to a different region of the map. To do this we’ll set up some simple HTML navigation links at the bottom of the page being generated via the iRule, and set up a switch structure to handle each URI the links pass back into the iRule, and use those to format the HTML appropriately so that we get the right Google charts call. That sounds more complicated than it is. Here’s what it looks like:

"/heatmap" {
 set chld ""
 set chd ""
 foreach country [table keys -subtable countries] {
 append chld $country
 append chd "[table lookup -subtable countries $country],"
 }
 set chd [string trimright $chd ","]
 HTTP::respond 200 content "<HTML><center>Here is your site's usage by Country: <img src='http://chart.apis.google.com/chart?cht=t&chd=&chs=440x220&chtm=world&chd=t:$chd&chld=$chld&chco=f5f5f5,edf0d4,6c9642,365e24,13390a' border='0'> Zoom to region: <a href='/asia'>Asia</a> | <a href='/africa'>Africa</a> | <a href='/europe'>Europe</a> | <a href='/middle_east'>Middle East</a> | <a href='/south_america'>South America</a> | <a href='/usa'>United States</a> | <a href='/heatmap'>World</a> <a href='/resetmap'>Reset All Counters</a></center></HTML>"

 "/asia" {
 set chld ""
 set chd ""
 foreach country [table keys -subtable countries] {
 append chld $country
 append chd "[table lookup -subtable countries $country],"
 }
 set chd [string trimright $chd ","]
 HTTP::respond 200 content "<HTML><center>Here is your site's usage by Country: <img src='http://chart.apis.google.com/chart?cht=t&chd=&chs=440x220&chtm=asia&chd=t:$chd&chld=$chld&chco=f5f5f5,edf0d4,6c9642,365e24,13390a' border='0'> Zoom to region: <a href='/asia'>Asia</a> | <a href='/africa'>Africa</a> | <a href='/europe'>Europe</a> | <a href='/middle_east'>Middle East</a> | <a href='/south_america'>South America</a> | <a href='/usa'>United States</a> | <a href='/heatmap'>World</a> <a href='/resetmap'>Reset All Counters</a></center></HTML>"

…

As you can see, we now have a world view map that shows the heat of each country, and we have individual links that we can click on along the bottom to take us to a zoom of each country/region to get a more specific look at the info there. As an example, let’s take a look at the data from Asia:

So we now have a nice little heatmapping application. It pulls up a world view of app traffic going to your site or app, it allows you to click around to the different regions of the world to get a more detailed view, and it even lets you re-set the data at will. I can hear some among you asking “What about the states, though?”. If I take away the state view of the US and give a world view, then I’m really trading one limitation for another. Ideally we’d be able to see both, right?

If I want to be able to give a detailed view on both the countries around the world and the states within the US, then I need to expand my data collection a bit. I need to collect both country codes for incoming requests and state abbreviations, where applicable. This means creating a second sub-table within the iRule, and issuing a second whereis per request coming in. Something like this should do:

      set cloc [whereis [IP::client_addr] country]
      set sloc [whereis [IP::client_addr] abbrev]
      if {[table incr -subtable countries -mustexist $cloc] eq ""} {
          table set -subtable countries $cloc 1 indefinite indefinite
      }
      if {[table incr -subtable states -mustexist $sloc] eq ""} {
          table set -subtable states $sloc 1 indefinite indefinite
      }

Above we’re using the cloc (country location) and sloc (state location) variables to simultaneously track both country codes and state abbreviations in separate sub tables within the iRule. This way we don’t mix up CA (Canada) and CA (California) or similar crossovers and throw our counts off. When doing this, don’t forget to update the resetmap case as well to empty both sub tables, not just one. This also means that we’ll need to slightly change the logic in the “usa” case as opposed to all of the other cases when doing a lookup. If the user wants to view the USA details, we need to do a subtable lookup on the states sub table, everything else uses the countries sub table. Not too horrible.

Okay, we now have heatmaps for all countries, all available zoom regions and a zoom to state level in the US, complete with some rudimentary HTML to make this feel like an application, not just a static image on a web page. Unfortunately, we also have around 140 lines of code, much of which is being repeated. There’s no sense in repeating that HTML over and over, or those logic statements doing the lookups and whatnot. So it’s time to take out the scalpel and start slicing and dicing, looking for unnecessary code.

I started with the HTML. There’s just no reason to repeat that HTML in every single switch case. So I set that in some static variables in the RULE_INIT section and did away with that all together in each switch case. Next, the actual iRules logic is identical if I want to view asia or africa or europe or anything other than the US. The only difference is the HTML changing one word to tell the API where to zoom in. Using a little extra “zoom” logic, I was able to cut down most of that repetitive code as well, by having all of the switch cases other than the USA fall through to the world view case, giving us just two chunks of iRules logic to deal with. Not including the extra variables and tidbits, the core of those two chunks of logic are:

      foreach country [table keys -subtable countries] {
        append chld $country
        append chd "[table lookup -subtable countries $country],"
      }


      foreach state [table keys -subtable states] {
        append chld $state
        append chd "[table lookup -subtable states $state],"
      }

Don’t stop there, though, there’s more to trim! With some more advanced trickery we can combine these two table lookups into a single piece of logic. When all is said and done, here is the final iRule trimmed down to fighting form with a single switch case handling the presentation of all the possible heatmaps generated by Google…pretty cool stuff:

when RULE_INIT {
set static::resp1 "<HTML><center>Here is your site's usage by Country: <img src='http://chart.apis.google.com/chart?cht=t&chd=&chs=440x220&chtm="
set static::resp2 "&chco=f5f5f5,edf0d4,6c9642,365e24,13390a' border='0'> Zoom to region: <a href='/asia'>Asia</a> | <a href='/africa'>Africa</a> | <a href='/europe'>Europe</a> | <a href='/middle_east'>Middle East</a> | <a href='/south_america'>South America</a> | <a href='/usa'>United States</a> | <a href='/heatmap'>World</a> <a href='/resetmap'>Reset All Counters</a></center></HTML>"
}

when HTTP_REQUEST {
switch -glob [string tolower [HTTP::uri]] {
 "/asia" -
 "/africa" -
 "/europe" -
 "/middle_east" -
 "/south_america" -
 "/usa" -
 "/world" -
 "/heatmap*" {
 set chld ""
 set chd ""
 set zoom [string map {"/" "" "heatmap" "world"} [HTTP::uri]]
## Configure the table query to be based on the countries subtable or the states subtable ##
 if {$zoom eq "usa"} {
 set region "states"
 } else {
 set region "countries"
 }
## Get a list of all states or countries and the associated count of requests from that area ##
 foreach rg [table keys -subtable $region] {
 append chld $rg
 append chd "[table lookup -subtable $region $rg],"
 }
 set chd [string trimright $chd ","]
## Send back the pre-formatted response, set in RULE_INIT, combined with the map zoom, list of areas, and request count ##
 HTTP::respond 200 content "${static::resp1}${zoom}&chd=t:${chd}&chld=${chld}${static::resp2}"
 }
 "/resetmap" {
 foreach country [table keys -subtable countries] {
 table delete -subtable countries $country
 }
 foreach state [table keys -subtable states] {
 table delete -subtable states $state
 }
 HTTP::respond 200 Content "<HTML><center> Table Cleared. <a href='/heatmap'>Return to Map</a></HTML>"
 }
 default {
## Look up country & state locations ##
 set cloc [whereis [IP::client_addr] country]
 set sloc [whereis [IP::client_addr] abbrev]
## If the IP doesn't resolve to anything, pick a random IP (useful for testing on private networks) ##
 if {($cloc eq "") and ($sloc eq "")} {
 set ip [expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }]
 set cloc [whereis $ip country]
 set sloc [whereis $ip abbrev]
 }
## Set Country ##
 if {[table incr -subtable countries -mustexist $cloc] eq ""} {
 table set -subtable countries $cloc 1 indefinite indefinite
 }
## Set State ##
 if {[table incr -subtable states -mustexist $sloc] eq ""} {
 table set -subtable states $sloc 1 indefinite indefinite
 }
 HTTP::respond 200 Content "Added"
 }
}
}

There we have it, an appropriately trimmed down and sleek application to provide worldwide or regional views of heatmaps showing traffic to your application, all generated 100% via iRules. Again, this couldn’t be done without the awesome geolocation abilities of LTM or the Google charting API or, of course, iRules. In the next installment we’ll dig even deeper to see how to turn this application into something even more valuable to those interested in what the users of your site or app are up to.

Published Jul 08, 2010

Version 1.0