Heatmaps, iRules Style: Part 1

When working with any type of traffic, metrics play an important role. Whether you’re dealing with web traffic, custom TCP application traffic, DNS requests…pretty much anything traveling over the wire, chances are at some point someone in your organization is going to want to know more than just “Yep, the traffic is flowing.”. This is where metrics come in. You’ll report on, well, something. Uptime, requests per x time period, utilization, or some other thing will need to be rolled up and reported to those interested parties so that they can make sure things are running properly, show off the great stats in presentations, or check the “metrics gathered” box off their quarterly objectives scorecard. Regardless, it’ll be up to you to gather that info.

I’ve been tinkering with a way to allow iRules to make that easier, and to allow those interested parties to see some usage statistics in a visually interesting, real-time manner, without adding much heavy lifting for you or your application. The idea is simple, create a heat map view of your HTTP requests, mapped to the locations around the United States (to start with). This will give you an idea which areas are most highly utilizing your application in a very easy on the eyes fashion. Best of all, of course, is that we’re going to generate this 100% with iRules. When all is said and done, you should end up with something that looks about like this.

This is possible for a couple reasons. First of all, iRules is awesome, and allows for all sorts of programmatic magic while processing traffic which we’ll get into below. Second though, and equally as important for this task is the somewhat new and completely amazing gelocation database that iRules has access to. By issuing a single, simple command via iRules I’m able to easily look up the originating location of any IP address. I make great use of that in this example, and it wouldn’t be possible without their data.

Okay, on to the solution. We need to be able to keep a counter in memory and increment it. We naturally want to be able to do this in the most efficient way possible to minimize drag on your LTM. So we need something that is “permanent”, global (not scoped to a given session), highly efficient, easily counted or incremented…hmm. If you’re like me, and running 10.1 or later, tables should be leaping to mind right about now. So let’s take a look at a relatively simple counter mechanism using tables:

when HTTP_REQUEST {
set loc [whereis [IP::client_addr] abbrev]
if {$loc eq ""} {
    set ip [expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }]
    set loc [whereis $ip abbrev]
}
if {[table incr -subtable states -mustexist $loc] eq ""} {
    table set -subtable states $loc 1 indefinite indefinite
}
}

So above we have a relatively simple block that looks up the location of every inbound IP address using the whereis command. For testing and demonstration purposes it’s then setting a random IP address in case you can’t look up the incoming IP. (All of my requests were on an internal network so none resolved and I wanted to test the functionality. You’ll want to remove that set ip set loc section if you aren’t looking for random IP info once you’re out of the testing or demo phase.) It then increments a subtable entry for the state returned in the lookup. It’s setting the subtable entry to have an indefinite timeout and lifetime the first time a request for that state comes in, otherwise just incrementing it by 1.

That works great and all, but now we have this indefinite lifetime entry, many of them probably, with data building up. We need some mechanism to reset the tables to 0, but I don’t want those to occur on a given lifetime or timeout, since I want it to be manual. I.E. I want the data to collect until you push the button to say “reset data”, effectively. So now we need another section to reset things on demand. What we’ll want to do is get a list of all the entries we’ve created and delete them. Fortunately, since we’ve been using a subtable, that’s trivial. We’ll just get a list of the keys in the subtable and delete each one. That would look something like this:

if {[HTTP::uri] starts_with "/resetmap"} {
foreach state [table keys -subtable states] {
table delete -subtable states $state
}
}

Now that we have a table counting every request that comes in, and a way to reset those tables as needed, we’re well on our way. All that’s left now is to actually output the heatmap that shows the visual representation of the compiled data we’re storing in the table shown above. To do that, we’ll query the table and get a list of all of the state abbreviation codes, along with their respective number of requests so far, and we’ll format that into a URL that is a call to Google’s charting API, using the chld and chd variables. Lastly, for presentation, we’ll wrap the whole mess in some very basic HTML (forgive me web-designers, for I am not one of you) that makes it feel a bit like an interface, as well as tweaking the default mapping color gradients a bit. The whole thing, when put together, looks like this:

when HTTP_REQUEST {
if {[HTTP::uri] starts_with "/heatmap"} {
 set chld ""
 set chd ""
 foreach state [table keys -subtable states] {
 append chld $state
 append chd "[table lookup -subtable states $state],"
 }
 set chd [string trimright $chd ","]
 HTTP::respond 200 content "<HTML><center>Here is your site's usage by state: <img src='http://chart.apis.google.com/chart?cht=t&chd=&chs=440x220&chtm=usa&chd=t:$chd&chld=$chld&chco=f5f5f5,edf0d4,6c9642,365e24,13390a' border='0'> <a href='/resetmap'>Reset Map</a></center></HTML>"
} elseif {[HTTP::uri] starts_with "/resetmap"} {
 foreach state [table keys -subtable states] {
 table delete -subtable states $state
 }
 HTTP::respond 200 Content "<HTML><center> Table Cleared. <a href='/heatmap'>Return to Map</a></HTML>"
} else {
 set loc [whereis [IP::client_addr] abbrev]
 if {$loc eq ""} {
 set ip [expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }]
 set loc [whereis $ip abbrev]
 }
 if {[table incr -subtable states -mustexist $loc] eq ""} {
 table set -subtable states $loc 1 indefinite indefinite
 }
}
}

There you have it, a fully functioning heatmapping application built entirely in iRules, with some much appreciated assistance from the geolocation database and Google’s charting API. This is, of course, an amazingly basic look at what this type of functionality can do. In Part 2 of this series I’ll be showing you how you can make this even more powerful and granular to give you a look at not just overall usage information but some more specific bits that will likely prove even more useful. Many thanks to Matt Cauthorn, one of the many truly outstanding engineers here at F5, for getting the juices flowing on this one and riffing on it with me to come up with the idea and start the planning.

Published Jun 29, 2010

Version 1.0