iControl Apps - #05 - Rate Based Statistics

One of the key features of iControl is the ability to monitor statistics of the various objects on the system.  Whether it's for capacity planning or auditing and billing purposes, the numbers behind how your applications are being utilized on the network are very important.  The iControl API allows for querying of statistics for most of the major objects including Virtual Servers, Pools, Pool Members, and Node Addresses.  This data is returned as counters but most often the desired presentation of this data is in the form of a rate such as bits or connections per second.  This article will show how to query the counter values and calculate rate based statistics for a given virtual server.

Usage

The arguments for this application are the address, username, and password of the BIG-IP along with optional argument for the specified virtual server.  This is declared in the top of the script with the following param statement.  There is also a Write-Usage function to display the arguments to the user.

param (
  $g_bigip = $null,
  $g_uid = $null,
  $g_pwd = $null,
  $g_virtual = $null
);

$g_gmt_offset = "";

Set-PSDebug -strict;

function Write-Usage()
{
  Write-Host "Usage: VirtualRates.ps1 host uid pwd [virtual_server]";
  exit;
}

Initialization

As is with all of my PowerShell scripts, the initialization component will look to see if the iControlSnapIn is loaded into the current PowerShell session.  If not, the Add-PSSnapIn Cmdlet is called to add the snapin into the runtime.  Then a call to the Initialize-F5.iControl cmdlet to setup the connection to the BIG-IP.  If this succeeds then success, then the rest of the optional arguments are processed.

If a virtual server name is not specifed, the Get-Virtuals function is called to list out all available virtual servers on the BIG-IP.

If a virtual server name is specified, the Get-VirtualRates function is called which continuously polls the BIG-IP in one second intervals.  During each interval it extracts the total bits, packets, and connections and then calculates rates based off of those numbers.

The code for this process is listed below.

function Do-Initialize()
{
  if ( (Get-PSSnapin | Where-Object { $_.Name -eq "iControlSnapIn"}) -eq $null )
  {
    Add-PSSnapIn iControlSnapIn
  }
  $success = Initialize-F5.iControl -HostName $g_bigip -Username $g_uid -Password $g_pwd;
 
  return $success;
}

if ( ($g_bigip -eq $null) -or ($g_uid -eq $null) -or ($g_pwd -eq $null) )
{
  Write-Usage;
}

if ( Do-Initialize )
{
  if ( $g_virtual -eq $null )
  {
    Get-Virtuals;
  }
  else
  {
    Get-VirtualRates $g_virtual;
  }
}
else
{
  Write-Error "ERROR: iControl subsystem not initialized"
}

Listing the Virtuals

The Get-Virtuals function will call the LocalLB.VirtualServer.get_list() method to query the list of virtual servers on the BIG-IP.  It will then loop over that list and write the results to the console.

function Get-Virtuals()
{
  $vs_list = (Get-F5.iControl).LocalLBVirtualServer.get_list();
  Write-Host "Available Virtual Servers:";
  foreach ($vs in $vs_list)
  {
    Write-Host "  $vs";
  }
}

Reporting Rates

The Get-VirtualRates function does most of the grunt work in this application.  It takes as a parameter a virtual server name and then enters a continuous loop.

In this loop, the LocalLB.VirtualServer.get_statistics() method is called.  The Common.TimeStamp structure from the result of that call is then converted to a .Net DateTime with the local Get-TimeFromTimeStamp function.

After the time conversion, the returned statistics for the specified virtual server and looks for the STATISTIC_CLIENT_SIDE_BYTES_IN, STATISTIC_CLIENT_SIDE_PACKETS_IN, and STATISTIC_CLIENT_SIDE_TOTAL_CONNECTIONS to use for calculations.

If this is the first time through the loop, there is no reference for a rate, so the values returned are stored in the b0, p0, and c0 variables respectively.  Note the Convert-To64Bit function used here.  We'll talk about that later but it basically takes the Common.UInt64 structure's high and low 32 bit values and turns it into a 64 bit number.

For all subsequent trips through the loop, a difference is calculated between the new and old counter values and these are stored in the b2, p2, and c2 variables.  The difference between the first and current poll is calculated by subtracting the t1 and t0 variables and this results in a .Net TimeSpan object that we extract the total seconds from and store that value in the sec variable.  For sanity, if the time difference rounds to zero, we'll set it to one to avoid a divide by zero error.

At this point another sanity check is performed to make sure that the difference in counters is not negative.  If it's negative, meaning that the last value is smaller than the first, that would mean that either the statistics have rolled over the 2^64 boundary or that they were manually reset.  If this is the case, then we'll throw this poll away and treat it as if the polling started over again.

Finally if we have a valid poll, the rates are calculated by total number over the interval by the time difference resulting in a X/sec rate.  The values for bits per second, packets per second, and connections per second are then presented to the console.

Function Get-VirtualRates()
{
  param($virtual_server);
 
  $bFirst = 1;
 
  $b0 = 0;
  $p0 = 0;
  $c0 = 0;
  $t0 = [DateTime]::Now;
 
  Write-Host "Rates for Virtual Server: $virtual_server"
 
  while (1)
  {
    $VirtualServerStatistics = (Get-F5.iControl).LocalLBVirtualServer.get_statistics( (, $virtual_server) );
    $t1 = Get-TimeFromTimeStamp $VirtualServerStatistics.time_stamp;
    $VirtualServerStatisticEntry = $VirtualServerStatistics.statistics[0];
    $Statistics = $VirtualServerStatisticEntry.statistics;

    foreach ($Statistic in $Statistics)
    {
      switch ($Statistic.type) {
        "STATISTIC_CLIENT_SIDE_BYTES_IN" {
          $b1 = Convert-To64Bit $Statistic.value.high $Statistic.value.low;
        }
        "STATISTIC_CLIENT_SIDE_PACKETS_IN" {
          $p1 = Convert-To64Bit $Statistic.value.high $Statistic.value.low;
        }
        "STATISTIC_CLIENT_SIDE_TOTAL_CONNECTIONS" {
          $c1 = Convert-To64Bit $Statistic.value.high $Statistic.value.low;
        }
      }
    }
   
    if ($bFirst -eq 1 )
    {
      $bFirst = 0;
      $b0 = $b1;
      $p0 = $p1;
      $c0 = $c1;
      $t0 = $t1;
    }
    else
    {
      $b2 = $b1 - $b0;
      $p2 = $p1 - $p0;
      $c2 = $c1 - $c0;
      $t2 = $t1 - $t0;
      $sec = $t2.Seconds;
      if  ( $sec -eq 0 ) { $sec = 1 }
     
      if ( ($b2 -lt 0) -or ($p2 -lt 0) -or ($c2 -lt 0) )
      {
        # either the counters increased past 2^64 or they were reset so start over.
        $bFirst = 1;
      }
      else
      {
        # Calculate rates
        $br = $b2/$sec;
        $pr = $p2/$sec;
        $cr = $c2/$sec;
       
        Write-Host "$br bps; $pr pps; $cr cps"
      }
    }
   
    Start-Sleep -s 1
  }
}

Converting Time

The Get-TimeFromTimeStamp function takes as input an iControl Common.TimeStamp structure and then creates a native .Net TimeStamp structure from those values and returns it to the calling function.

function Get-TimeFromTimeStamp()
{
  param ($TimeStamp);
  $dt = new-object -typename System.DateTime
  $dt = $dt.AddYears($TimeStamp.year-1)
    .AddMonths($TimeStamp.month-1)
    .AddDays($TimeStamp.day-1)
    .AddHours($TimeStamp.hour)
    .AddMinutes($TimeStamp.minute)
    .AddSeconds($TimeStamp.second);
  return $dt;
}

Converting to 64 bit

The iControl value returned for statistics is the Common.UInt64 structure which has two 32 bit values for the low and high 32 bits of the resulting 64 bit number.  Unfortunately, at the time of this article, PowerShell does not support bitwise shifting so we can't create the 64 bit number natively in PowerShell.  Never fear!  Since PowerShell supports .Net and .Net supports inline compiling and running of .Net code, we can inline compile the C# needed to do the conversion.  I found a handy PowerShell script by Lee Holmes/Joel Bennett providing support for inline C# in PowerShell.  I converted this script to a powershell function.  The Convert-To64Bit function builds the C# 32 bit parts to 64 bit conversion code, calls the Invoke-Inline function with it, and returns the result.

function Convert-To64Bit()
{
  param($high, $low);
  return Invoke-Inline "result.Add((Convert.ToUInt64($high)<<32) | (Convert.ToUInt64($low)));"
}

Running C# inline in PowerShell

As mentioned above, I took the Invoke-Inline powershell script from Lee Holmes/Joel Bennett and converted it into a inline function for this application.  I'm not going to go into the details of this function but in short it takes as input some C# code, builds a cached compiled version of it, executes the supplied code, and then returns the results - if there are any.  Very cool!

function Invoke-Inline()
{
  param(
    [string[]] $code,
    [object[]] $arguments,
    [string[]] $reference = @()
  )

  ## Stores a cache of generated inline objects.  If this library is dot-sourced
  ## from a script, these objects go away when the script exits.
  if(-not (Test-Path Variable:\inlineCode.Cache))
  {
    ${GLOBAL:inlineCode.Cache} = @{}
  }

  $using = $null;
  $source = $null;
  if($code.length -eq 1) {
    $source = $code[0]
  } elseif($code.Length -eq 2){
    $using = $code[0]
    $source = $code[1]
  } else {
    Write-Error "You have to pass some code, or this won't do anything ..."
  }

  ## un-nesting magic (why do I need this?)
  $params = @()
  foreach($arg in $arguments) { $params += $arg }
  $arguments = $params

  ## The main function to execute inline C#.  
  ## Pass the argument to the function as a strongly-typed variable.  They will 
  ## be available from C# code as the Object variable, "arg".
  ## Any values assigned to the "returnValue" object by the C# code will be 
  ## returned to MSH as a return value.

  ## See if the code has already been compiled and cached
  $cachedObject = ${inlineCode.Cache}[$source]
  #Write-Verbose "Type: $($arguments[0].GetType())"

  ## The code has not been compiled or cached
  if($cachedObject -eq $null)
  {
    $codeToCompile =
@"
    using System;
    using System.Collections.Generic;
    $using

    public class InlineRunner
    {
      public List<object> Invoke(Object[] args)
      {
        List<object> result = new List<object>();

        $source

        if( result.Count > 0 ) {
            return result;
        } else {
            return null;
        }
      }
    }
"@
    Write-Verbose $codeToCompile

    ## Obtains an ICodeCompiler from a CodeDomProvider class.
    $provider = New-Object Microsoft.CSharp.CSharpCodeProvider

    ## Get the location for System.Management.Automation DLL
    $dllName = [PsObject].Assembly.Location

    ## Configure the compiler parameters
    $compilerParameters = New-Object System.CodeDom.Compiler.CompilerParameters

    $assemblies = @("System.dll", $dllName)
    $compilerParameters.ReferencedAssemblies.AddRange($assemblies)
    $compilerParameters.ReferencedAssemblies.AddRange($reference)
    $compilerParameters.IncludeDebugInformation = $true
    $compilerParameters.GenerateInMemory = $true

    ## Invokes compilation. 
    $compilerResults =
      $provider.CompileAssemblyFromSource($compilerParameters, $codeToCompile)

    ## Write any errors if generated.        
    if($compilerResults.Errors.Count -gt 0)
    {
      $errorLines = ""
      foreach($error in $compilerResults.Errors)
      {
        $errorLines += "`n`t" + $error.Line + ":`t" + $error.ErrorText
      }
      Write-Error $errorLines
    }
    ## There were no errors.  Store the resulting object in the object
    ## cache.
    else
    {
      ${inlineCode.Cache}[$source] =
        $compilerResults.CompiledAssembly.CreateInstance("InlineRunner")
    }

    $cachedObject = ${inlineCode.Cache}[$source]
   }

   Write-Verbose "Argument $arguments`n`n$cachedObject"
   ## Finally invoke the C# code
   if($cachedObject -ne $null)
   {
     return $cachedObject.Invoke($arguments)
   }
}

Running the program

The following command will not supply the virtual server name so a list of available virtual servers will be returned.

PS C:\> .\VirtualRates.ps1 theboss admin admin
Available Virtual Servers:
  xpbert-ftp
  xpbert-http
  xpbert-ssh
  xpbert-telnet

For this example, I'll monitor the xpbert-http virtual server.  For this test, the virtual is initially idle and then I ran apachebench against the virtual with 10000 total/50 concurrent connections to the default page on the webserver.  You'll see starting at poll #3 the cps starts increasing and then when the bit counts start kicking in the bits per second (bps) and packets per second (pps) start increasing.  When the apachebench process completes, you will see that the rates will start decreasing over time.  Since there is no traffic on the virtual it's fairly obvious that since the time is still increasing but the statistic over that interval remains the same, then the rate will decrease.

PS C:\> .\VirtualRates.ps1 theboss admin admin xpbert-http
Rates for Virtual Server: xpbert-http
0 bps; 0 pps; 0 cps
0 bps; 0 pps; 0 cps
0 bps; 0 pps; 77 cps
0 bps; 0 pps; 232 cps
32398.8 bps; 411.6 pps; 350.8 cps
207088.166666667 bps; 2625.5 pps; 453.5 cps
177504.142857143 bps; 2250.42857142857 pps; 528.285714285714 cps
203525.625 bps; 2585.625 pps; 570.5 cps
216543 bps; 2751 pps; 554.9 cps
246497.363636364 bps; 3131.54545454545 pps; 550 cps
298047.083333333 bps; 3783.75 pps; 554.083333333333 cps
255468.928571429 bps; 3243.21428571429 pps; 545.357142857143 cps
253460 bps; 3220 pps; 567.2 cps
271884.0625 bps; 3454.0625 pps; 593.75 cps
287913.705882353 bps; 3657.70588235294 pps; 588.235294117647 cps
301427.611111111 bps; 3829.38888888889 pps; 555.555555555556 cps
275500 bps; 3500 pps; 500 cps
262380.952380952 bps; 3333.33333333333 pps; 476.190476190476 cps
250454.545454545 bps; 3181.81818181818 pps; 454.545454545455 cps
239565.217391304 bps; 3043.47826086957 pps; 434.782608695652 cps
229583.333333333 bps; 2916.66666666667 pps; 416.666666666667 cps
220400 bps; 2800 pps; 400 cps
211923.076923077 bps; 2692.30769230769 pps; 384.615384615385 cps
204074.074074074 bps; 2592.59259259259 pps; 370.37037037037 cps
196785.714285714 bps; 2500 pps; 357.142857142857 cps
190000 bps; 2413.79310344828 pps; 344.827586206897 cps
183666.666666667 bps; 2333.33333333333 pps; 333.333333333333 cps
177741.935483871 bps; 2258.06451612903 pps; 322.58064516129 cps
172187.5 bps; 2187.5 pps; 312.5 cps
166969.696969697 bps; 2121.21212121212 pps; 303.030303030303 cps

Conclusion

In this article, I've presented a way to query a virtual server for it's throughput and connection statistics and use that data to calculate rates based on the time interval between polls. This logic can easily be extended to any other object on the BIG-IP such as pools, pool members, or node addresses.

For a full version of this script, check out the wiki entry for PsRateBasedStatistics in the iControl CodeShare.