Scatter Plotting Response Times With iRules And Google Charts

We’ve presented several articles in the past utilizing the Google Charts API with iControl and iRules.  A few of these include

I realized we have yet to present an example of a Scatter Chart, so I set to put together one that did just that.

The Chart

To get your attention, I’ll change the format of my tech tip a little bit by presenting the output first, and then I’ll go over the iRule I used to generate it.

The data collected for this example is calculated for a page request to an application page.  The HTTP response code (200, 302, 404, etc), the size of the response payload (page size), and the duration (time it took for the backend server to process and return the page).  The output of the iRule consists of two charts and a few control commands.

The first chart represents the response time for a page to be processed by the backend server against the response code from the server.   The size of the dots in the chart represent the size in bytes of the response.  And the second chart plots the response time on the X-Axis with the size of the payload.

image_282620.jpg

The iRule

Now we’ll dig into the iRule used to store the data and build the report and embedded charts.

Data Collection

The data collection component of this iRule is fairly straightforward.  in the HTTP_REQUEST event for incoming page requests, we store the system time in milliseconds into the t1 variable.  When the response comes back from the server, we store the current time in the t2 variable.  A difference is then calculated representing the duration of the request.  The HTTP status code and the response size, indicated in the “Content-Length” response header determined as well.  The three of these values are combined together with a semicolon separator and then inserted into the data subtable.    I also create a subtable to store the various response codes for a reason you’ll see below.

All the stored data is inserted with a default 180 second timeout, forcing the data to be purged from the session table after 3 minutes.  This ensures that the data table won’t grow too big and we can stay within the 2000 character URL limit in the resulting graphs.

<span id="lnum1" style="color: #606060;">   1:</span> when HTTP_REQUEST {
<span id="lnum2" style="color: #606060;">   2:</span>   set t1 [clock clicks -milliseconds];
<span id="lnum3" style="color: #606060;">   3:</span> }
<span id="lnum4" style="color: #606060;">   4:</span> 
<span id="lnum5" style="color: #606060;">   5:</span> when HTTP_RESPONSE {
<span id="lnum6" style="color: #606060;">   6:</span>   set t2 [clock clicks -milliseconds];
<span id="lnum7" style="color: #606060;">   7:</span>   set duration [expr {$t2 - $t1}];
<span id="lnum8" style="color: #606060;">   8:</span>   set code [HTTP::status];
<span id="lnum9" style="color: #606060;">   9:</span>   set clen [HTTP::header Content-Length];
<span id="lnum10" style="color: #606060;">  10:</span>
<span id="lnum11" style="color: #606060;">  11:</span>   # Only monitor application traffic
<span id="lnum12" style="color: #606060;">  12:</span>   <span style="color: #0000ff;">if</span> { [HTTP::header Content-Type] starts_with <span style="color: #006080;">"text/html"</span> } {
<span id="lnum13" style="color: #606060;">  13:</span>
<span id="lnum14" style="color: #606060;">  14:</span>     # Only allow $MAX_KEYS <span style="color: #0000ff;">in</span> table.
<span id="lnum15" style="color: #606060;">  15:</span>
<span id="lnum16" style="color: #606060;">  16:</span>     set data <span style="color: #006080;">"${code}:${clen}:${duration}"</span>;
<span id="lnum17" style="color: #606060;">  17:</span>     <span style="color: #0000ff;">if</span> { [table incr -subtable $DATA_TABLE -mustexist $data] eq <span style="color: #006080;">""</span>} {
<span id="lnum18" style="color: #606060;">  18:</span>       table set -subtable $DATA_TABLE $data 1 $TIMEOUT $LIFETIME;
<span id="lnum19" style="color: #606060;">  19:</span>     }
<span id="lnum20" style="color: #606060;">  20:</span>
<span id="lnum21" style="color: #606060;">  21:</span>     <span style="color: #0000ff;">if</span> { [table incr -subtable $CODES_TABLE -mustexist $code] eq <span style="color: #006080;">""</span> } {
<span id="lnum22" style="color: #606060;">  22:</span>       table set -subtable $CODES_TABLE $code 1 $TIMEOUT $LIFETIME;
<span id="lnum23" style="color: #606060;">  23:</span>     }
<span id="lnum24" style="color: #606060;">  24:</span>   }
<span id="lnum25" style="color: #606060;">  25:</span> }

Global Settings

I’ve got several global variables.  These most likely should either be hard coded or stored in global scope in the RULE_INIT method, but I included them in the HTTP_REQUEST event for ease of reading in the article.  The size of the charts can be customized as well as the data timeout and the subtable names.

<span id="lnum1" style="color: #606060;">   1:</span> set CHART_WIDTH 400
<span id="lnum2" style="color: #606060;">   2:</span> set CHART_HEIGHT 325
<span id="lnum3" style="color: #606060;">   3:</span> set MAX_KEYS 300;
<span id="lnum4" style="color: #606060;">   4:</span> set TIMEOUT 180;
<span id="lnum5" style="color: #606060;">   5:</span> set LIFETIME indefinite;
<span id="lnum6" style="color: #606060;">   6:</span> set DATA_TABLE <span style="color: #006080;">"scatterchart"</span>;
<span id="lnum7" style="color: #606060;">   7:</span> set CODES_TABLE <span style="color: #006080;">"scatterchartcodes"</span>;

Handling Report Requests

The main page request for this iRule application, is processed when the user requests the “/scatterchart” url on the virtual that this iRule is applied to.  A simple HTML page is generated containing images with links back to the iRule for the charts described below.  A few other commands are added with links to refresh the page, reset the data, and to popup a HTML table with all the raw data.  Each of those commands will be described below.

<span id="lnum1" style="color: #606060;">   1:</span> <span style="color: #0000ff;">switch</span> [HTTP::uri] {
<span id="lnum2" style="color: #606060;">   2:</span>   <span style="color: #006080;">"/scatterchart"</span> {
<span id="lnum3" style="color: #606060;">   3:</span>     set content <span style="color: #006080;">"<html><head><title>iRule Scatter Chart</title></head><body><center>"</span>;
<span id="lnum4" style="color: #606060;">   4:</span>     append content <span style="color: #006080;">"<img src='/scatterchart/t_by_c' width=${CHART_WIDTH} height='${CHART_HEIGHT}'>"</span>;
<span id="lnum5" style="color: #606060;">   5:</span>     append content <span style="color: #006080;">"<img src='/scatterchart/d_by_s' width=${CHART_WIDTH} height='${CHART_HEIGHT}'>"</span>;
<span id="lnum6" style="color: #606060;">   6:</span>     append content <span style="color: #006080;">"<br/><a href='/scatterchart'>RELOAD</a>"</span>;
<span id="lnum7" style="color: #606060;">   7:</span>     append content <span style="color: #006080;">" | <a href='/scatterchart/reset'>RESET</a>"</span>;
<span id="lnum8" style="color: #606060;">   8:</span>     append content <span style="color: #006080;">" | <a href='/scatterchart/data' target='dtwindow'>DATA</a>"</span>;
<span id="lnum9" style="color: #606060;">   9:</span>     append content <span style="color: #006080;">"</center></html>"</span>;
<span id="lnum10" style="color: #606060;">  10:</span>     HTTP::respond 200 Content $content;
<span id="lnum11" style="color: #606060;">  11:</span>   }
<span id="lnum12" style="color: #606060;">  12:</span> }

The Time By Status Code Chart

The first chart plots the duration by the status code.  This section of code stores the data series for the chart in the d_x (X-Value), d_y (Y-Value), and d_s (Size of dot) variables.  The records in the data subtable are looped over with the foreach command.  I have a sanity check with the CUR_KEY variable to make sure the URL doesn’t go beyond the 2000 character limit.  The status code, length, and duration are extracted from the record with the getfield command and the various data variables are appended to.

In the data collection portion of the iRule, I created a subtable to contain the various status codes.  With this data, then build custom Y-Axis labels for the status codes returned.   Since we are plotting by the code and the code is a number that equals the Y-height on the chart, the chart label (chxl) and point (chxp) values are the same.

The URL to the Google Chart is then generated from the data and a HTTP::redirect is issued to the browser to the actual URL at Google’s servers.

<span id="lnum1" style="color: #606060;">   1:</span> <span style="color: #0000ff;">switch</span> [HTTP::uri] {
<span id="lnum2" style="color: #606060;">   2:</span>   <span style="color: #006080;">"/scatterchart/t_by_c"</span> {
<span id="lnum3" style="color: #606060;">   3:</span>     # Time by response code
<span id="lnum4" style="color: #606060;">   4:</span>     set d_x <span style="color: #006080;">""</span>;
<span id="lnum5" style="color: #606060;">   5:</span>     set d_y <span style="color: #006080;">""</span>;
<span id="lnum6" style="color: #606060;">   6:</span>     set d_s <span style="color: #006080;">""</span>;
<span id="lnum7" style="color: #606060;">   7:</span>
<span id="lnum8" style="color: #606060;">   8:</span>     set CUR_KEY 0;
<span id="lnum9" style="color: #606060;">   9:</span>
<span id="lnum10" style="color: #606060;">  10:</span>     <span style="color: #0000ff;">foreach</span> key [table keys -subtable $DATA_TABLE] {
<span id="lnum11" style="color: #606060;">  11:</span>       incr CUR_KEY;
<span id="lnum12" style="color: #606060;">  12:</span>       <span style="color: #0000ff;">if</span> { $CUR_KEY > $MAX_KEYS } { <span style="color: #0000ff;">break</span>; }
<span id="lnum13" style="color: #606060;">  13:</span>
<span id="lnum14" style="color: #606060;">  14:</span>       set     code [getfield $key <span style="color: #006080;">":"</span> 1];
<span id="lnum15" style="color: #606060;">  15:</span>       set      len [getfield $key <span style="color: #006080;">":"</span> 2];
<span id="lnum16" style="color: #606060;">  16:</span>       set duration [getfield $key <span style="color: #006080;">":"</span> 3];
<span id="lnum17" style="color: #606060;">  17:</span> 
<span id="lnum18" style="color: #606060;">  18:</span>       set len [expr $len / 75];
<span id="lnum19" style="color: #606060;">  19:</span>       <span style="color: #0000ff;">if</span> { $len < 300 } { set len 300; }
<span id="lnum20" style="color: #606060;">  20:</span> 
<span id="lnum21" style="color: #606060;">  21:</span>       <span style="color: #0000ff;">if</span> { $d_x != <span style="color: #006080;">""</span> } { append d_x <span style="color: #006080;">","</span>; }
<span id="lnum22" style="color: #606060;">  22:</span>       append d_x $duration
<span id="lnum23" style="color: #606060;">  23:</span>       <span style="color: #0000ff;">if</span> { $d_y != <span style="color: #006080;">""</span> } { append d_y <span style="color: #006080;">","</span>; }
<span id="lnum24" style="color: #606060;">  24:</span>       append d_y $code;
<span id="lnum25" style="color: #606060;">  25:</span>       <span style="color: #0000ff;">if</span> { $d_s != <span style="color: #006080;">""</span> } { append d_s <span style="color: #006080;">","</span>; }
<span id="lnum26" style="color: #606060;">  26:</span>       append d_s $len;
<span id="lnum27" style="color: #606060;">  27:</span>     }
<span id="lnum28" style="color: #606060;">  28:</span>
<span id="lnum29" style="color: #606060;">  29:</span>     set CHXL <span style="color: #006080;">""</span>;
<span id="lnum30" style="color: #606060;">  30:</span>     set CHXP <span style="color: #006080;">""</span>;
<span id="lnum31" style="color: #606060;">  31:</span>
<span id="lnum32" style="color: #606060;">  32:</span>     # Build Y-Axis labels with the HTTP status codes
<span id="lnum33" style="color: #606060;">  33:</span>     <span style="color: #0000ff;">foreach</span> code [table keys -subtable $CODES_TABLE] {
<span id="lnum34" style="color: #606060;">  34:</span>       <span style="color: #0000ff;">if</span> { $CHXL != <span style="color: #006080;">""</span> } { append CHXL <span style="color: #006080;">"|"</span> }
<span id="lnum35" style="color: #606060;">  35:</span>       append CHXL $code;
<span id="lnum36" style="color: #606060;">  36:</span>       <span style="color: #0000ff;">if</span> { $CHXP != <span style="color: #006080;">""</span> } { append CHXP <span style="color: #006080;">","</span> }
<span id="lnum37" style="color: #606060;">  37:</span>       append CHXP $code;
<span id="lnum38" style="color: #606060;">  38:</span>     }
<span id="lnum39" style="color: #606060;">  39:</span>
<span id="lnum40" style="color: #606060;">  40:</span>     set XMIN 0;
<span id="lnum41" style="color: #606060;">  41:</span>     set XMAX 1000;
<span id="lnum42" style="color: #606060;">  42:</span>     set YMIN 0;
<span id="lnum43" style="color: #606060;">  43:</span>     set YMAX 500;
<span id="lnum44" style="color: #606060;">  44:</span>     set SMAX 1000;
<span id="lnum45" style="color: #606060;">  45:</span>
<span id="lnum46" style="color: #606060;">  46:</span>     set chartUrl <span style="color: #006080;">"http://1.chart.apis.google.com/chart?cht=s"</span>;
<span id="lnum47" style="color: #606060;">  47:</span>     append chartUrl <span style="color: #006080;">"&chtt=Web+Requests+Time+By+Status+Code"</span>;
<span id="lnum48" style="color: #606060;">  48:</span>     append chartUrl <span style="color: #006080;">"&chxt=x,y&chs=${CHART_WIDTH}x${CHART_HEIGHT}"</span>;
<span id="lnum49" style="color: #606060;">  49:</span>     append chartUrl <span style="color: #006080;">"&chco=F1F12F|989866|000000|224499|AA0033"</span>;
<span id="lnum50" style="color: #606060;">  50:</span>     append chartUrl <span style="color: #006080;">"&chxr=0,${XMIN},${XMAX}|1,${YMIN},${YMAX}"</span>;
<span id="lnum51" style="color: #606060;">  51:</span>     append chartUrl <span style="color: #006080;">"&chds=0,${XMAX},0,${YMAX},0,${SMAX}"</span>;
<span id="lnum52" style="color: #606060;">  52:</span>     append chartUrl <span style="color: #006080;">"&chxl=1:${CHXL}"</span>;
<span id="lnum53" style="color: #606060;">  53:</span>     append chartUrl <span style="color: #006080;">"&chxp=1,${CHXP}"</span>;
<span id="lnum54" style="color: #606060;">  54:</span>     append chartUrl <span style="color: #006080;">"&chg=0,10"</span>;
<span id="lnum55" style="color: #606060;">  55:</span>     append chartUrl <span style="color: #006080;">"&chd=t:${d_x}|${d_y}|${d_s}"</span>
<span id="lnum56" style="color: #606060;">  56:</span>
<span id="lnum57" style="color: #606060;">  57:</span>     HTTP::redirect $chartUrl;
<span id="lnum58" style="color: #606060;">  58:</span>   }
<span id="lnum59" style="color: #606060;">  59:</span> }

The Duration By Size Chart

The generation for this chart is very similar to the previous one, except that we’ve changed the data values a bit.  This time the X-Axis represents the duration and the Y-Axis the size of the response.  I’ve added code to grow the X and Y Axis maximum values on the chart if the data exceeds the current maximum values.  The records in the session table are iterated upon as in the previous section of code and the chart URL is generated and redirected to the client.

<span id="lnum1" style="color: #606060;">   1:</span> <span style="color: #0000ff;">switch</span> [HTTP::uri] {
<span id="lnum2" style="color: #606060;">   2:</span>   <span style="color: #006080;">"/scatterchart/d_by_s"</span> {
<span id="lnum3" style="color: #606060;">   3:</span>     # Duration by Size
<span id="lnum4" style="color: #606060;">   4:</span>     set d_x <span style="color: #006080;">""</span>;
<span id="lnum5" style="color: #606060;">   5:</span>     set d_y <span style="color: #006080;">""</span>;
<span id="lnum6" style="color: #606060;">   6:</span>     set d_s <span style="color: #006080;">""</span>;
<span id="lnum7" style="color: #606060;">   7:</span>     set rows <span style="color: #006080;">""</span>;
<span id="lnum8" style="color: #606060;">   8:</span>
<span id="lnum9" style="color: #606060;">   9:</span>     set XMIN 0;
<span id="lnum10" style="color: #606060;">  10:</span>     set XMAX 1000;
<span id="lnum11" style="color: #606060;">  11:</span>     set YMIN 0;
<span id="lnum12" style="color: #606060;">  12:</span>     set YMAX 500;
<span id="lnum13" style="color: #606060;">  13:</span>     set SMAX 100;
<span id="lnum14" style="color: #606060;">  14:</span>
<span id="lnum15" style="color: #606060;">  15:</span>     set CUR_KEY 0;
<span id="lnum16" style="color: #606060;">  16:</span>
<span id="lnum17" style="color: #606060;">  17:</span>     <span style="color: #0000ff;">foreach</span> key [table keys -subtable $DATA_TABLE] {
<span id="lnum18" style="color: #606060;">  18:</span>       incr CUR_KEY;
<span id="lnum19" style="color: #606060;">  19:</span>       <span style="color: #0000ff;">if</span> { $CUR_KEY > $MAX_KEYS } { <span style="color: #0000ff;">break</span>; }
<span id="lnum20" style="color: #606060;">  20:</span>
<span id="lnum21" style="color: #606060;">  21:</span>       set     code [getfield $key <span style="color: #006080;">":"</span> 1];
<span id="lnum22" style="color: #606060;">  22:</span>       set      len [getfield $key <span style="color: #006080;">":"</span> 2];
<span id="lnum23" style="color: #606060;">  23:</span>       set duration [getfield $key <span style="color: #006080;">":"</span> 3];
<span id="lnum24" style="color: #606060;">  24:</span> 
<span id="lnum25" style="color: #606060;">  25:</span>       <span style="color: #0000ff;">if</span> { $d_x != <span style="color: #006080;">""</span> } { append d_x <span style="color: #006080;">","</span>; }
<span id="lnum26" style="color: #606060;">  26:</span>       append d_x $duration
<span id="lnum27" style="color: #606060;">  27:</span>       <span style="color: #0000ff;">if</span> { $duration > $XMAX } { set XMAX $duration; }
<span id="lnum28" style="color: #606060;">  28:</span>
<span id="lnum29" style="color: #606060;">  29:</span>       <span style="color: #0000ff;">if</span> { $d_y != <span style="color: #006080;">""</span> } { append d_y <span style="color: #006080;">","</span>; }
<span id="lnum30" style="color: #606060;">  30:</span>       append d_y $len;
<span id="lnum31" style="color: #606060;">  31:</span>       <span style="color: #0000ff;">if</span> { $len > $YMAX } { set YMAX $len; }
<span id="lnum32" style="color: #606060;">  32:</span>
<span id="lnum33" style="color: #606060;">  33:</span>       <span style="color: #0000ff;">if</span> { $d_s != <span style="color: #006080;">""</span> } { append d_s <span style="color: #006080;">","</span>; }
<span id="lnum34" style="color: #606060;">  34:</span>       append d_s 100;
<span id="lnum35" style="color: #606060;">  35:</span>     }
<span id="lnum36" style="color: #606060;">  36:</span>
<span id="lnum37" style="color: #606060;">  37:</span>     set chartUrl <span style="color: #006080;">"http://0.chart.apis.google.com/chart?cht=s"</span>;
<span id="lnum38" style="color: #606060;">  38:</span>     append chartUrl <span style="color: #006080;">"&chtt=Web+Requests+Duration+By+Size"</span>;
<span id="lnum39" style="color: #606060;">  39:</span>     append chartUrl <span style="color: #006080;">"&chxt=x,y&chs=${CHART_WIDTH}x${CHART_HEIGHT}"</span>;
<span id="lnum40" style="color: #606060;">  40:</span>     append chartUrl <span style="color: #006080;">"&chco=F1F12F|989866|000000|224499|AA0033"</span>;
<span id="lnum41" style="color: #606060;">  41:</span>     append chartUrl <span style="color: #006080;">"&chxr=0,${XMIN},${XMAX}|1,${YMIN},${YMAX}"</span>;
<span id="lnum42" style="color: #606060;">  42:</span>     append chartUrl <span style="color: #006080;">"&chds=0,${XMAX},0,${YMAX},0,${SMAX}"</span>;
<span id="lnum43" style="color: #606060;">  43:</span>     append chartUrl <span style="color: #006080;">"&chg=0,10"</span>;
<span id="lnum44" style="color: #606060;">  44:</span>     append chartUrl <span style="color: #006080;">"&chd=t:${d_x}|${d_y}|${d_s}"</span>
<span id="lnum45" style="color: #606060;">  45:</span>
<span id="lnum46" style="color: #606060;">  46:</span>     HTTP::redirect $chartUrl;
<span id="lnum47" style="color: #606060;">  47:</span>   }
<span id="lnum48" style="color: #606060;">  48:</span> }

A Raw Data Table

For debugging purposes, I threw in a command to generate a HTML table containing all the records in the table.   A 3-column table is then returned to the browser with the Code, Length, and Duration values in the data table.

<span id="lnum1" style="color: #606060;">   1:</span> <span style="color: #0000ff;">switch</span> [HTTP::uri] {
<span id="lnum2" style="color: #606060;">   2:</span>   <span style="color: #006080;">"/scatterchart/data"</span> {
<span id="lnum3" style="color: #606060;">   3:</span>     set content <span style="color: #006080;">"<html><head><title>Scatter Chart raw data</title></head><body><center>"</span>;
<span id="lnum4" style="color: #606060;">   4:</span>     append content <span style="color: #006080;">"<a href='/scatterchart/data'>RELOAD</a><br/>"</span>;
<span id="lnum5" style="color: #606060;">   5:</span>     append content <span style="color: #006080;">"<table border='1'><tr><th>Code</th><th>Length</th><th>Duration (ms)</th></tr>"</span>;
<span id="lnum6" style="color: #606060;">   6:</span>
<span id="lnum7" style="color: #606060;">   7:</span>     <span style="color: #0000ff;">foreach</span> key [table keys -subtable $DATA_TABLE] {
<span id="lnum8" style="color: #606060;">   8:</span>       set     code [getfield $key <span style="color: #006080;">":"</span> 1];
<span id="lnum9" style="color: #606060;">   9:</span>       set      len [getfield $key <span style="color: #006080;">":"</span> 2];
<span id="lnum10" style="color: #606060;">  10:</span>       set duration [getfield $key <span style="color: #006080;">":"</span> 3];
<span id="lnum11" style="color: #606060;">  11:</span>       append content <span style="color: #006080;">"<tr><td>${code}</td><td>${len}</td><td>${duration}</td></tr>"</span>;
<span id="lnum12" style="color: #606060;">  12:</span>     }
<span id="lnum13" style="color: #606060;">  13:</span>
<span id="lnum14" style="color: #606060;">  14:</span>     append content <span style="color: #006080;">"</table><br/><a href='/scatterchart/data'>RELOAD</a></center></body></html>"</span>;
<span id="lnum15" style="color: #606060;">  15:</span>
<span id="lnum16" style="color: #606060;">  16:</span>     HTTP::respond 200 content $content;
<span id="lnum17" style="color: #606060;">  17:</span>   }
<span id="lnum18" style="color: #606060;">  18:</span> }

Data Reset

If you are just too impatient to wait 3 minutes for the records to reset, or you are running this on a highly active site and you want to clear out all the data at any given time, I’ve included a command to reset the subtables.  Passing in the URI “/scatterchart/reset” will erase all entries in the subtables and let you start clean.

<span id="lnum1" style="color: #606060;">   1:</span> <span style="color: #0000ff;">switch</span> [HTTP::uri] {
<span id="lnum2" style="color: #606060;">   2:</span>   <span style="color: #006080;">"/scatterchart/reset"</span> {
<span id="lnum3" style="color: #606060;">   3:</span>     table delete -subtable $DATA_TABLE -all;
<span id="lnum4" style="color: #606060;">   4:</span>     table delete -subtable $CODES_TABLE -all;
<span id="lnum5" style="color: #606060;">   5:</span>     HTTP::redirect <span style="color: #006080;">"/scatterchart"</span>;
<span id="lnum6" style="color: #606060;">   6:</span>   }
<span id="lnum7" style="color: #606060;">   7:</span> }

Get The Full Source

The source for this iRule can be found in the iRule CodeShare under the topic ScatterChartingResponseTimes.

Related Articles on DevCentral

Technorati Tags: Reporting, iRules, table, Google, Chart, Heatmap, Joe Pruitt