Forum Discussion

Nimbostratus

Aug 26, 2010

Heat Maps irule by Colin.

Hi,

Thanks to Colin Walker for the great heatmap's irule.

I have tried to implement this irule but run into some problems and not sure where to look..Below is what I see in the log files..

If I have the IP randomization part of the code left in then the site (app, not the heatmaps site) loads every other time. Also I had commented out the

HTTP::respond 200 Content "Added"

Else I see Added on the real site instead of the homepage.

Please help.

Thanks,

-jay

Aug 26 17:09:42 local/tmm err tmm[5151]: 01220001:3: TCL error: heatmap_irule - Illegal value (line 1) invoked from within "table incr -subtable countries -mustexist $cloc" ("default" arm line 12) invoked from within "switch -glob [string tolower [HTTP::uri]] { "/asia" - "/africa" - "/europe" - "/middle_east" - "/south_americ..."

Aug 26 17:09:42 local/tmm1 err tmm1[5152]: 01220001:3: TCL error: heatmap_irule - Illegal value (line 1) invoked from within "table incr -subtable countries -mustexist $cloc" ("default" arm line 12) invoked from within "switch -glob [string tolower [HTTP::uri]] { "/asia" - "/africa" - "/europe" - "/middle_east" - "/south_americ..."

Aug 26 17:09:42 local/tmm1 info tmm1[5152]: 01220001:6: Per-invocation log rate exceeded; throttling.

Aug 26 17:09:42 local/tmm info tmm[5151]: 01220001:6: Per-invocation log rate exceeded; throttling.

Aug 26 17:09:54 local/tmm1 info tmm1[5152]: 01220001:6: Resuming log processing at this invocation; held 8 messages.

Nimbostratus

Ive tried this rule out as well and had to edit it to get it to work correctly. It seems the author didnt actually test the final rule. In testing the recently edited rule works fantastic and amazed my CEO; in production however, it is a different story. Within 1 min of being deployed onto our production site the Google Chart API started denying traffic. The bigIP LTM CPU ramped up to 100% and another min after that and the whole bigip LTM device crashed completely. Upon restart of the device we had to wait almost an hour for the TMM to finish cleanup before traffic started serving again. Here is the modified rule that caused the trouble.

when RULE_INIT {
      Configure static portions of the HTML response for the heatmap pages
      set static::resp1 ""
   }

   
when HTTP_REQUEST timing on {
if {  [HTTP::uri] != "/"}  { 
     switch -glob [string tolower [HTTP::uri]] {
       "/asia*" - 
       "/africa*" - 
       "/europe*" - 
       "/middle_east*" - 
       "/south_america*" - 
       "/usa*" -
       "/world*" - 
       "/heatmap*" {
         set chld ""  
         set chd ""
         set zoom ""
         set zoomURL ""  
         set regions ""  
         set urlTotal 0  
         set regionTotal 0 
     Split apart the zoom region from the filter URL in the request  
         set zoom [getfield [string map {"/" "" "heatmap" "world"} [HTTP::uri]] "?" 1]
         set zoomURL [getfield [string map {"/" "" "heatmap" "world"} [HTTP::uri]] "?" 2]
     Get a list of all states or countries, applying the URL filter where necessary
     and retrieve the associated count of requests from that area to that URL
       First step through the mytables table, which is a pointer table referencing all subtables with counter values in them
         foreach mysub [table keys -subtable mytables] {
       Next determine whether to search state or country tables
           if {$zoom eq "usa"} {
             if {$mysub starts_with "state:"} { 
       For each state sub table step through each key, which will be a URL, and count the request to that URL.
       This is also where URL filtering is applied if applicable
               foreach myurl [table keys -subtable $mysub] {
                 if {$zoomURL ne ""} { 
                   if {$myurl eq $zoomURL} {  
                     append chld "[getfield $mysub ":" 2]"
                     append chd "[table lookup -subtable $mysub $myurl],"
                     set urlTotal [table lookup -subtable $mysub $myurl]
                   }
                 } else {
                   append chld "[getfield $mysub ":" 2]"
                   append chd "[table lookup -subtable $mysub $myurl],"  
                   set urlTotal [table lookup -subtable $mysub $myurl]
                 }
                 set regionTotal [expr $regionTotal + $urlTotal]  
                 set urlTotal 0  
               }  
               append regions "[getfield $mysub ":" 2] : $regionTotal
"
               set regionTotal 0
             } 
           } else {
             if {$mysub starts_with "country:"} { 
               foreach myurl [table keys -subtable $mysub] {
                 if {$zoomURL ne ""} { 
                   if {$myurl eq $zoomURL} {  
                     append chld "[getfield $mysub ":" 2]"
                     append chd "[table lookup -subtable $mysub $myurl],"
                     set urlTotal [table lookup -subtable $mysub $myurl]
                   }
                   set regionTotal [expr $regionTotal + $urlTotal]  
                   set urlTotal 0  
                 } else {
                   append chld "[getfield $mysub ":" 2]"
                   append chd "[table lookup -subtable $mysub $myurl],"  
                   set urlTotal [table lookup -subtable $mysub $myurl]
                 }
                   set regionTotal [expr $regionTotal + $urlTotal]  
                     set urlTotal 0    
               }  
               append regions "[getfield $mysub ":" 2] : $regionTotal
"
               set regionTotal 0
             } 
           }
         }
 
 
     Send back the pre-formatted response, set in RULE_INIT, combined with the map zoom, list of areas, and request count
         set chd [string trimright $chd ","] 
       First loop through the trackingurls class to get a list of all URLs to be tracked and format HTML around them for links
         set filters ""  
         foreach mytrackingurl [class names trackingurls] {
           append filters " | "
         }  
         set filters [string trimright $filters " | "]   
  
       Combine the above generated HTML with the static HTML in RULE INIT and respond to the client
        HTTP::respond 200 content "${static::resp1}${regions}${static::resp2}${zoom}&chd=t:${chd}&chld=${chld}${static::resp3}Filter by URL:  | $filters $static::resp4"
       }
      
      "/resetmap" {
        foreach pointertable [table keys -subtable mytables] {
          foreach entry [table keys -subtable $pointertable] {
            table delete -subtable $pointertable $entry
          }  
        } 
        foreach pointerentry [table keys -subtable mytables] {
          table delete -subtable mytables $pointerentry
        }  
        HTTP::respond 200 Content "
Table Cleared."
      }
   
      default {
   Look up country & state locations 
        set cloc [whereis [IP::client_addr] country]
        set sloc [whereis [IP::client_addr] abbrev]
      
    If the IP doesn't resolve to anything, pick a random IP (useful for testing on private networks)Uncomment for test.
        if {($cloc eq "") and ($sloc eq "")} { 
          set ip [expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }]
          set cloc [whereis $ip country]
          set sloc [whereis $ip abbrev]
          if {($cloc eq "") or ($sloc eq "")} {  
              set cloc "US"
            set sloc "WA"
          }
        }
   
    Strip slashes from URI to allow easy queries
        set friendlyURL [string map {/ ""} [HTTP::uri]]
    Create a new table named country:location or state:location
        if {[table incr -subtable country:$cloc -mustexist $friendlyURL] eq ""} {
          table set -subtable country:$cloc $friendlyURL 1 indefinite indefinite
        }  
    Update the mytables pointer table with the new country or state table name  
        if {[table incr -subtable mytables -mustexist country:$cloc] eq ""} {
          table set -subtable mytables country:$cloc 1 indefinite indefinite
        }  
    Same as above for states, not countries.
        if {$cloc eq "US"} {  
          if {[table incr -subtable state:$sloc -mustexist $friendlyURL] eq ""} {
              table set -subtable state:$sloc $friendlyURL 1 indefinite indefinite
          }  
          if {[table incr -subtable mytables -mustexist state:$sloc] eq ""} {
             table set -subtable mytables state:$sloc 1 indefinite indefinite
          }    
        }
Uncomment if you want to see what state you are coming from.
        HTTP::respond 200 Content "Added - Country: $cloc State: $sloc"
  }   
 }
}
}
 site usage Heatmap:Connections per Region:
"
      set static::resp2  "
Click to zoom to region: |  |  |  |  |  | "
      set static::resp4 "

jay_41157
Nimbostratus
Feb 11, 2011
Jason are you planning on doing more testing and using in prod?

Thanks,

Jay
hooleylist
Cirrostratus
Feb 11, 2011
Can you post the recent entries from /var/log/ltm before the failure occurred?

Aaron

jay_41157

Nimbostratus

Feb 11, 2011

HI Aaron,

Actually except for the URL section the irule works now, I started agin.

here it is:

when RULE_INIT {

   Configure static portions of the HTML response for the heatmap pages

   set static::resp1 "
Table Cleared.
 "

     }

  

     default {

   Look up country & state locations 

       set cloc [whereis [IP::client_addr] country]

       set sloc [whereis [IP::client_addr] abbrev]

     

   If the IP doesn't resolve to anything, pick a random IP (useful for testing on private networks)

       if {($cloc eq "") and ($sloc eq "")} { 

         set ip [expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }]

         set cloc [whereis $ip country]

         set sloc [whereis $ip abbrev]

         if {($cloc eq "") or ($sloc eq "")} {  

             set cloc "US"

           set sloc "WA"

         }

       }

  

   Strip slashes from URI to allow easy queries

       set friendlyURL [string map {/ ""} [HTTP::uri]]

   Create a new table named country:location or state:location
        if {[table incr -subtable country:$cloc -mustexist $friendlyURL] eq ""} {

         table set -subtable country:$cloc $friendlyURL 1 indefinite indefinite 
}   

   Update the mytables pointer table with the new country or state table name  

       if {[table incr -subtable mytables -mustexist country:$cloc] eq ""} {

         table set -subtable mytables country:$cloc 1 indefinite indefinite

       }  

   Same as above for states, not countries.

       if {$cloc eq "US"} {  

         if {[table incr -subtable state:$sloc -mustexist $friendlyURL] eq ""} {

             table set -subtable state:$sloc $friendlyURL 1 indefinite indefinite

         }  

         if {[table incr -subtable mytables -mustexist state:$sloc] eq ""} {

             table set -subtable mytables state:$sloc 1 indefinite indefinite

         }   
       }    

       HTTP::respond 200 Content "Added - Country: $cloc State: $sloc"

     }

   }

 }
  
  
 Here are the logs: 
 Feb 11 17:20:35 local/tmm1 err tmm1[7379]: 01220001:3: TCL error: heatmaps_part3_irule  - Illegal value (line 1)     invoked from within "table incr -subtable country:$cloc -mustexist $friendlyURL"     ("default" arm line 38)     invoked from within "switch -glob [string tolower [HTTP::uri]] {         "/asia*" -        "/africa*" -        "/europe*" -        "/middle_east*" -        "/south_america..." 
 Feb 11 17:20:35 local/tmm err tmm[7378]: 01220001:3: TCL error: heatmaps_part3_irule  - Illegal value (line 1)     invoked from within "table incr -subtable country:$cloc -mustexist $friendlyURL"     ("default" arm line 38)     invoked from within "switch -glob [string tolower [HTTP::uri]] {         "/asia*" -        "/africa*" -        "/europe*" -        "/middle_east*" -        "/south_america..." 
 Feb 11 17:20:35 local/tmm1 err tmm1[7379]: 01220001:3: TCL error: heatmaps_part3_irule  - Illegal value (line 1)     invoked from within "table incr -subtable country:$cloc -mustexist $friendlyURL"     ("default" arm line 38)     invoked from within "switch -glob [string tolower [HTTP::uri]] {         "/asia*" -        "/africa*" -        "/europe*" -        "/middle_east*" -        "/south_america..." 
 Feb 11 17:20:35 local/tmm err tmm[7378]: 01220001:3: TCL error: heatmaps_part3_irule  - Illegal value (line 1)     invoked from within "table incr -subtable country:$cloc -mustexist $friendlyURL"     ("default" arm line 38)     invoked from within "switch -glob [string tolower [HTTP::uri]] {         "/asia*" -        "/africa*" -        "/europe*" -        "/middle_east*" -        "/south_america..." 
  
  
  
 Thanks,  
 JayHere is your site's usage:\
Connections per Region:
"

   set static::resp2  "
Zoom to region:\
 |  |  |  | \ |  | "

   set static::resp4 ""

 }

  

 when HTTP_REQUEST timing on {

   switch -glob [string tolower [HTTP::uri]] {  

     "/asia*" - 

     "/africa*" - 

     "/europe*" - 

     "/middle_east*" - 

     "/south_america*" - 

     "/usa*" -

     "/world*" - 

     "/m_a_p_s_t_a_t_s*" {

       set chld ""  

       set chd ""

       set zoom ""

       set zoomURL ""  

       set regions ""  

       set urlTotal 0  

       set regionTotal 0

   Split apart the zoom region from the filter URL in the request  

       set zoom [getfield [string map {"/" "" "m_a_p_s_t_a_t_s" "world"} [HTTP::uri]] "?" 1]

       set zoomURL [getfield [string map {"/" "" "m_a_p_s_t_a_t_s" "world"} [HTTP::uri]] "?" 2]

   Get a list of all states or countries, applying the URL filter where necessary

   and retrieve the associated count of requests from that area to that URL

     First step through the mytables table, which is a pointer table referencing all subtables with counter values in them

       foreach mysub [table keys -subtable mytables] {

     Next determine whether to search state or country tables

         if {$zoom eq "usa"} {

           if {$mysub starts_with "state:"} { 

     For each state sub table step through each key, which will be a URL, and count the request to that URL.

     This is also where URL filtering is applied if applicable

             foreach myurl [table keys -subtable $mysub] {

               if {$zoomURL ne ""} { 

                 if {$myurl eq $zoomURL} {  

                   append chld "[getfield $mysub ":" 2]"

                   append chd "[table lookup -subtable $mysub $myurl],"

                   set urlTotal [table lookup -subtable $mysub $myurl]

                 }

               } else {

                 append chld "[getfield $mysub ":" 2]"

                 append chd "[table lookup -subtable $mysub $myurl],"  

                 set urlTotal [table lookup -subtable $mysub $myurl]

               }

               set regionTotal [expr $regionTotal + $urlTotal]  

               set urlTotal 0  

             }  

             append regions "[getfield $mysub ":" 2] : $regionTotal"

             set regionTotal 0

           } 

         } else {

           if {$mysub starts_with "country:"} { 

             foreach myurl [table keys -subtable $mysub] {

               if {$zoomURL ne ""} { 

                 if {$myurl eq $zoomURL} {  

                   append chld "[getfield $mysub ":" 2]"

                   append chd "[table lookup -subtable $mysub $myurl],"

                   set urlTotal [table lookup -subtable $mysub $myurl]

                 }

                 set regionTotal [expr $regionTotal + $urlTotal]  

                 set urlTotal 0  

               } else {

                 append chld "[getfield $mysub ":" 2]"

                 append chd "[table lookup -subtable $mysub $myurl],"  

                 set urlTotal [table lookup -subtable $mysub $myurl]

               }

                 set regionTotal [expr $regionTotal + $urlTotal]  

                 set urlTotal 0    

             }  

             append regions "[getfield $mysub ":" 2] : $regionTotal"

             set regionTotal 0

           } 

         }

       }

  

   Send back the pre-formatted response, set in RULE_INIT, combined with the map zoom, list of areas, and request count

       set chd [string trimright $chd ","] 

     First loop through the trackingurls class to get a list of all URLs to be tracked and format HTML around them for links

       set filters ""  
   
       foreach mytrackingurl [class names trackingurls] {

         append filters " | "

       }  

       set filters [string trimright $filters " | "]   

  

     Combine the above generated HTML with the static HTML in RULE INIT and respond to the client

      HTTP::respond 200 content "${static::resp1}${regions}${static::resp2}${zoom}&chd=t:${chd}&chld=${chld}${static::resp3} \

      Filter by URL:  | $filters\

      $static::resp4"

     }

     

     "/resetmap" {

       foreach pointertable [table keys -subtable mytables] {

         foreach entry [table keys -subtable $pointertable] {

           table delete -subtable $pointertable $entry

         }  

       } 

       foreach pointerentry [table keys -subtable mytables] {

         table delete -subtable mytables $pointerentry

       } 

       HTTP::respond 200 Content "

hooleylist
Cirrostratus
Feb 11, 2011
I haven't read through the iRule fully enough to understand all of the logic. But I think the immediate cause of the runtime TCL error is that $friendlyURL is null. It looks like this would happen if the URI was just forward slashes (one or more), as those are getting removed when saving the $friendlyURL variable here:

Strip slashes from URI to allow easy queries

set friendlyURL [string map {/ ""} [HTTP::uri]]

Aaron
spark_86682
Historic F5 Account
Feb 12, 2011
If you try to use "table incr" on a table entry which has a string for a value (and not a number), then that will give you an "Illegal value" error as you're getting there. For instance:

table set "key" "value" table incr "key"

will give you that error, because "value" can't be converted to a number, so it can't be incremented. That could be what you're seeing.
jay_41157
Nimbostratus
Feb 14, 2011
Posted By hoolio on 02/11/2011 02:51 PM

I haven't read through the iRule fully enough to understand all of the logic. But I think the immediate cause of the runtime TCL error is that $friendlyURL is null. It looks like this would happen if the URI was just forward slashes (one or more), as those are getting removed when saving the $friendlyURL variable here:

Strip slashes from URI to allow easy queries

set friendlyURL [string map {/ ""} [HTTP::uri]]

Aaron

Hi Aaron,

I added a check if its root url only then I strip the /'s else I leave them alone.

Strip slashes from URI to allow easy queries if its no root level

if { not ([HTTP::uri] equals "/" ) } {

set friendlyURL [string map {/ ""} [HTTP::uri]]

}

set friendlyURL [HTTP::uri]

This seems to be working. No errors in the logs I will request the developers to do load testing on this app.

The next question I have is regarding url classifications... I have created a data group called trackingurls in this data group I have a few URI's that I want to categorize stats by...

on the stats page I do not see any stats based on the url's all 0's any thoughts?

Thanks,

Jay
nadavush_65504
Nimbostratus
Mar 05, 2012
Thanks for the share. it was insightful I love my heatmap suite.