Heatmaps, iRules Style: Part 4, Meaningful numbers
All right, so far I’ve built a heatmap of the US, expanded it to show the entire world with zoom capability, and allowed you to sort the views of those different dynamically generated maps by URLs of your own choosing. So far so good, but a very important piece of this puzzle is still missing. So you show off the fancy new heatmaps your LTM can generate and people are oohing and ahhing, but then someone asks, “So what does this dark green mean? How many requests have we actually seen to url123 from California?”. At which point you say, “uhhhhhh….” and suddenly look far less impressive. Well we can’t have that, can we?
In this fourth installment of my Heatmaps, iRules Style series I’ll show you how to add a little bit of extra HTML and counter logic to the already nifty heatmap code so that you can display the actual data being collected in an easy to consume format so that you can back up the pretty pictures with hard numbers to make everyone happy.
First, we’re using the exact code from Part 3 of the series (found here) as our starting point. All we’re really trying to do is get a list of all states or countries and the total number of connections we’ve received for that region. Keep in mind, though, that we’re not storing the connection count on a per state or per country basis. We’re using the URL for sorting as the key in the tables, so we need to throw together a sum of the requests to each URL for a given region (state or country). To do that we’ll just add in some logic that gets the numberic value of requests for each URL at the time when we’re appending things to the chld and chd for the API call, then total those for each subtable, since we have one subtable per region anyway (what, you thought that was arbitrary?) That looks like this:
1: append chld "[getfield $mysub ":" 2]"
2: append chd "[table lookup -subtable $mysub $myurl],"
3: set urlTotal [table lookup -subtable $mysub $myurl]
4: }5: set regionTotal [expr $regionTotal + $urlTotal]
6: set urlTotal 0
That will give us the number of connections for each region, now let’s pair that with the region name. We already have the logic we need to grab the name of the region from the append chld line, so all we need to do is pair it with the new regionTotal we’ve just created:
1: append regions "[getfield $mysub ":" 2] : $regionTotal<br>"
2: set regionTotal 0
So now if we’ve done things right we should have an entry that looks like “WA : 589” if the region in question is Washington State, for example. If we add this logic in the right places in the loops that build the API call we’ll have a nice list of the data we’re looking for. Now all we need to do is tweak the HTML a bit to add some structure to the whole thing and allow us to display this neat new info. To do so, the RULE_INIT section changes to:
1: set static::resp1 "<HTML><table width='100%' border='0'><tr><td></td><td><center><font size=5>Here is your site's \
2: usage:</font></td></tr><tr><td align='center' rowspan='5'><b><u>Connections per Region:</u></b><br><br>"3: set static::resp2 "</td><td><center><img src='http://chart.apis.google.com/chart?cht=t&chd=&chs=440x220&chtm="
4: set static::resp3 "&chco=f5f5f5,edf0d4,6c9642,365e24,13390a' border='0'></center></td></tr><td><center><br><br>\
5: Zoom to region: <a href='/asia'>Asia</a> | <a href='/africa'>Africa</a> | <a href='/europe'>Europe</a> | \ 6: <a href='/middle_east'>Middle East</a> | <a href='/south_america'>South America</a> | <a href='/usa'>United States</a> | \ 7: <a href='/heatmap'>World</a><br><br></td></tr><tr><td><center>"8: set static::resp4 "<tr><td><center><br><br><br><a href='/resetmap'>Reset All Counters</a></center></td></tr><tr></tr></HTML>"
And the HTTP::respond portion changes to:
1: HTTP::respond 200 content "${static::resp1}${regions}${static::resp2}${zoom}&chd=t:${chd}&chld=${chld}${static::resp3} \
2: Filter by URL: <a href='/$zoom'>All URLs</a> | $filters\ 3: $static::resp4"Note that all that needed to be added to the HTTP::respond code was a single variable, hence some of the craziness in the RULE_INIT section. I’m trying to keep things in the functional part of the code as simple as I can within reason.
That’s it, we should now have a list of all the regions and their total number of connections. The only real caveat is that there’s no good way to only display the countries listed in the zoom region (I.E. only African sub-regions when viewing Africa or only countries in the Asian via when viewing Asia) since google is the one dictating which countries show up in each zoom view, so you get a list of all countries recorded on all views. The cool part, however, is that because of the way the logic is structured and the way the counters were implemented your totals for region update based on the filter URL!
So now you can view a pretty map of any region of the world you want, filter that view based on a URL of your choosing to see how a given portion of your application is doing in that region, and have a key of sorts to see the actual number of connections in addition to the pretty picture. Neat, if you ask me, and pretty easy to boot, all things considered.
Pictures of the output and complete source below, as always. That wraps up my immediate ideas for the Heatmap series, but if you have any cool ideas or requests, let me know and I might just crank out a fifth installment or beyond.
1: when RULE_INIT {
2: ## Configure static portions of the HTML response for the heatmap pages
3: set static::resp1 "<HTML><table width='100%' border='0'><tr><td></td><td><center><font size=5>Here is your site's usage:\
4: </font></td></tr><tr><td align='center' rowspan='5'><b><u>Connections per Region:</u></b><br><br>"5: set static::resp2 "</td><td><center><img src='http://chart.apis.google.com/chart?cht=t&chd=&chs=440x220&chtm="
6: set static::resp3 "&chco=f5f5f5,edf0d4,6c9642,365e24,13390a' border='0'></center></td></tr><td><center><br><br>Zoom to region:\
7: <a href='/asia'>Asia</a> | <a href='/africa'>Africa</a> | <a href='/europe'>Europe</a> | <a href='/middle_east'>Middle East</a> | \ 8: <a href='/south_america'>South America</a> | <a href='/usa'>United States</a> | <a href='/heatmap'>World</a><br><br></td></tr><tr><td><center>"9: set static::resp4 "<tr><td><center><br><br><br><a href='/resetmap'>Reset All Counters</a></center></td></tr><tr></tr></HTML>"
10: } 11: 12: when HTTP_REQUEST timing on {
13: switch -glob [string tolower [HTTP::uri]] {
14: "/asia*" -
15: "/africa*" -
16: "/europe*" -
17: "/middle_east*" -
18: "/south_america*" -
19: "/usa*" -
20: "/world*" -
21: "/heatmap*" {
22: set chld ""
23: set chd ""
24: set zoom ""
25: set zoomURL ""
26: set regions ""
27: set urlTotal 0
28: set regionTotal 0
29: ## Split apart the zoom region from the filter URL in the request
30: set zoom [getfield [string map {"/" "" "heatmap" "world"} [HTTP::uri]] "?" 1]
31: set zoomURL [getfield [string map {"/" "" "heatmap" "world"} [HTTP::uri]] "?" 2]
32: ## Get a list of all states or countries, applying the URL filter where necessary
33: ## and retrieve the associated count of requests from that area to that URL
34: ## First step through the mytables table, which is a pointer table referencing all subtables with counter values in them
35: foreach mysub [table keys -subtable mytables] {
36: ## Next determine whether to search state or country tables
37: if {$zoom eq "usa"} {
38: if {$mysub starts_with "state:"} {
39: ## For each state sub table step through each key, which will be a URL, and count the request to that URL.
40: ## This is also where URL filtering is applied if applicable
41: foreach myurl [table keys -subtable $mysub] {
42: if {$zoomURL ne ""} {
43: if {$myurl eq $zoomURL} {
44: append chld "[getfield $mysub ":" 2]"
45: append chd "[table lookup -subtable $mysub $myurl],"
46: set urlTotal [table lookup -subtable $mysub $myurl]
47: }48: } else {
49: append chld "[getfield $mysub ":" 2]"
50: append chd "[table lookup -subtable $mysub $myurl],"
51: set urlTotal [table lookup -subtable $mysub $myurl]
52: }53: set regionTotal [expr $regionTotal + $urlTotal]
54: set urlTotal 0
55: } 56: append regions "[getfield $mysub ":" 2] : $regionTotal<br>"
57: set regionTotal 0
58: } 59: } else {
60: if {$mysub starts_with "country:"} {
61: foreach myurl [table keys -subtable $mysub] {
62: if {$zoomURL ne ""} {
63: if {$myurl eq $zoomURL} {
64: append chld "[getfield $mysub ":" 2]"
65: append chd "[table lookup -subtable $mysub $myurl],"
66: set urlTotal [table lookup -subtable $mysub $myurl]
67: }68: set regionTotal [expr $regionTotal + $urlTotal]
69: set urlTotal 0
70: } else {
71: append chld "[getfield $mysub ":" 2]"
72: append chd "[table lookup -subtable $mysub $myurl],"
73: set urlTotal [table lookup -subtable $mysub $myurl]
74: }75: set regionTotal [expr $regionTotal + $urlTotal]
76: set urlTotal 0
77: } 78: append regions "[getfield $mysub ":" 2] : $regionTotal<br>"
79: set regionTotal 0
80: } 81: } 82: } 83: 84: ## Send back the pre-formatted response, set in RULE_INIT, combined with the map zoom, list of areas, and request count
85: set chd [string trimright $chd ","]
86: ## First loop through the trackingurls class to get a list of all URLs to be tracked and format HTML around them for links
87: set filters ""
88: foreach mytrackingurl [class names trackingurls] {
89: append filters "<a href='${zoom}?${mytrackingurl}'>${mytrackingurl}</a> | "
90: } 91: set filters [string trimright $filters " | "]
92: 93: ## Combine the above generated HTML with the static HTML in RULE INIT and respond to the client
94: HTTP::respond 200 content "${static::resp1}${regions}${static::resp2}${zoom}&chd=t:${chd}&chld=${chld}${static::resp3} \
95: Filter by URL: <a href='/$zoom'>All URLs</a> | $filters\ 96: $static::resp4" 97: } 98: 99: "/resetmap" {
100: foreach pointertable [table keys -subtable mytables] {
101: foreach entry [table keys -subtable $pointertable] {
102: table delete -subtable $pointertable $entry
103: } 104: } 105: foreach pointerentry [table keys -subtable mytables] {
106: table delete -subtable mytables $pointerentry
107: } 108: HTTP::respond 200 Content "<HTML><center><br><br><br><br><br><br>Table Cleared.<br><br><br> <a href='/heatmap'>Return to Map</a></HTML>"
109: } 110: 111: default {
112: ## Look up country & state locations
113: set cloc [whereis [IP::client_addr] country]
114: set sloc [whereis [IP::client_addr] abbrev]
115: 116: ## If the IP doesn't resolve to anything, pick a random IP (useful for testing on private networks)
117: if {($cloc eq "") and ($sloc eq "")} {
118: set ip [expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }]
119: set cloc [whereis $ip country]
120: set sloc [whereis $ip abbrev]
121: if {($cloc eq "") or ($sloc eq "")} {
122: set cloc "US"
123: set sloc "WA"
124: } 125: } 126: 127: ## Strip slashes from URI to allow easy queries
128: set friendlyURL [string map {/ ""} [HTTP::uri]]
129: ## Create a new table named country:location or state:location
130: if {[table incr -subtable country:$cloc -mustexist $friendlyURL] eq ""} {
131: table set -subtable country:$cloc $friendlyURL 1 indefinite indefinite
132: } 133: ## Update the mytables pointer table with the new country or state table name
134: if {[table incr -subtable mytables -mustexist country:$cloc] eq ""} {
135: table set -subtable mytables country:$cloc 1 indefinite indefinite
136: } 137: ## Same as above for states, not countries.
138: if {$cloc eq "US"} {
139: if {[table incr -subtable state:$sloc -mustexist $friendlyURL] eq ""} {
140: table set -subtable state:$sloc $friendlyURL 1 indefinite indefinite
141: } 142: if {[table incr -subtable mytables -mustexist state:$sloc] eq ""} {
143: table set -subtable mytables state:$sloc 1 indefinite indefinite
144: } 145: } 146: HTTP::respond 200 Content "Added - Country: $cloc State: $sloc"
147: } 148: } 149: }
Related Articles
- Heatmaps, iRules Style: Part 1 > DevCentral > F5 DevCentral > Tech ...
- Heatmaps, iRules Style: Part2 > DevCentral > F5 DevCentral > Tech Tips
- Heatmaps, iRules Style: Part 3 – URL Filtering > DevCentral ...
- iRules Pointer Tables Demystified > DevCentral > F5 DevCentral ...
- iRule Editor - Editing iRules > DevCentral > F5 DevCentral > Tech Tips
Published Sep 21, 2010
Version 1.0
Haarith_DevarajNimbostratus
This is really cool. Thanks 🙂
Can you think of a way we can do this on the GTM.
That would be a fantastic add for this.
http://devcentral.f5.com/Community/GroupDetails/tabid/1082223/asg/52/afv/topic/aft/1173024/aff/31/showtab/groupforums/Default.aspx
lot of people using f5 gtm are trying to come up with something on the irule where we can tag not just the ldns locations ( we can do that through icontrol by pulling stats), but something which can show where the requests went too.
Cant seem to do tables or even a catch on the irule side.- Haarith_DevarajThis is really cool. Thanks 🙂
Can you think of a way we can do this on the GTM.
That would be a fantastic add for this.
http://devcentral.f5.com/Community/GroupDetails/tabid/1082223/asg/52/afv/topic/aft/1173024/aff/31/showtab/groupforums/Default.aspx
lot of people using f5 gtm are trying to come up with something on the irule where we can tag not just the ldns locations ( we can do that through icontrol by pulling stats), but something which can show where the requests went too.
Cant seem to do tables or even a catch on the irule side. 
Randy_Black_758when adding the rule, then adding it to a pool, I get the following:
Rule [/Common/heatmap] error: Unable to find value_list (trackingurls) referenced at line 85: [class names trackingurls]
The code is intact from the above article.....- JRahm
Admin
you need to add a datagroup called trackingurls - Randy_Black_758Well, now I'm in over my head. Back to the books.
Thanks! - JRahmA datagroup can be added in same location as iRules in the gui, Colin discusses the need/reason for the class in part 3: https://devcentral.f5.com/Tutorials/TechTips/tabid/63/articleType/ArticleView/articleId/1086368/Heatmaps-iRules-Style-Part-3-ampndash-URL-Filtering.aspx
- Randy_Black_758Thanks for the help.
Not sure how to ask or what the appropriate next step is. I have the iRule built in the F5. I used the code from it's entirety posted on part4. I have also created the datagroup. From the level of knowledge of I had, I creating thi, and this is what the results are:
"ltm data-group internal /Common/trackingurls {
type string
}"
I applied this to a less than heavy traffic VIP and let it run for awhile, I'm not sure of two things at this point:
If it is working - stats didn't indicate if it was, I did get counter increments in stats but I have no idea if that was a result of adding that iRule to the VIP.
Second, where do I use the data, does the rule produce a url that I use with the maps api....? If anyone can help, that will be awesome, I'll be over here wondering aimlessly in the weeds.
Thanks again,
Randy
