iControl 101 - #11 - Performance Graphs

The BIG-IP stores history of certain types of data for reporting purposes.  The Performance item in the Overview tab shows the report graphs for this performance data.  The management GUI gives you the options to customize the graph interval but it stops short of giving you access to the raw data behind the graphs.  Never fear, the System.Statistics interface contains methods to query the report types and extract the CSV data behind them.  You can even select the start and end times as well as the poll intervals.  This article will discuss the performance graph methods and how you can query the data behind them and build a chart of your own.

Initialization

This article uses PowerShell and the iControl Cmdlets for PowerShell as the client environment for querying the data.  The following setup will be required for the examples contained in this article.

PS> Add-PSSnapIn iControlSnapIn
PS> Initialize-F5.iControl -Hostname bigip_address -Username bigip_user -Password bigip_pass
PS> $SystemStats = (Get-F5.iControl).SystemStatistics

Now that that is taken care of, let's dive right in.  In the System.Statistics interface, there are two methods that get you to the performance graph data.  The first is the get_performance_graph_list() method.  This method takes no parameters and returns a list of structures containing each graph name, title, and description.

PS> $SystemStats.get_performance_graph_list()

graph_name                              graph_title                             graph_description
----------                              -----------                             -----------------
memory                                  Memory Used                             Memory Used
activecons                              Active Connections                      Active Connections
newcons                                 New Connections                         New Connections
throughput                              Throughput                              Throughput
httprequests                            HTTP Requests                           HTTP Requests
ramcache                                RAM Cache Utilization                   RAM Cache Utilization
detailactcons1                          Active Connections                      Active Connections
detailactcons2                          Active PVA Connections                  Active PVA Connections
detailactcons3                          Active SSL Connections                  Active SSL Connections
detailnewcons1                          Total New Connections                   Total New Connections
detailnewcons2                          New PVA Connections                     New PVA Connections
detailnewcons3                          New ClientSSL Profile Connections       New ClientSSL Profile Connections
detailnewcons4                          New Accepts/Connects                    New Accepts/Connects
detailthroughput1                       Client-side Throughput                  Client-side Throughput
detailthroughput2                       Server-side Throughput                  Server-side Throughput
detailthroughput3                       HTTP Compression Rate                   HTTP Compression Rate
SSLTPSGraph                             SSL Transactions/Sec                    SSL Transactions/Sec
GTMGraph                                GTM Performance                         GTM Requests and Resolutions
GTMrequests                             GTM Requests                            GTM Requests
GTMresolutions                          GTM Resolutions                         GTM Resolutions
GTMpersisted                            GTM Resolutions Persisted               GTM Resolutions Persisted
GTMret2dns                              GTM Resolutions Returned to DNS         GTM Resolutions Returned to DNS
detailcpu0                              CPU Utilization                         CPU Usage
detailcpu1                              CPU Utilization                         CPU Usage
CPU                                     CPU Utilization                         CPU Usage
detailtmm0                              TMM Utilization                         TMM Usage
TMM                                     TMM Utilization                         TMM CPU Utilization

Creating a Report

Ok, so you've now got the graph names, it's time to move on to accessing the data.  The method you'll want is the get_performance_graph_csv_statistics() method.  This method takes an array of PerformanceStatisticQuery structures containing the query parameters and returns an array of PerformanceGraphDataCSV structures, one for each input query.  The following code illustrates how to make a simple query.  The object_name corresponds to the graph_name in the get_performance_graph_list() method. 

The start_time and end_time allow you to control what the data range is.  Values of 0 (the default) will return the entire result set.  If the user specifies a start_time, end_time, and interval that does not exactly match the corresponding value used within the database, the database will attempt to use to closest time or interval as requested. The actual values used will be returned to the user on output. For querying purposes, the start_time can be specified as: 0: in which case by default, it means 24 hours ago. N: where N represents the number of seconds since Jan 1, 1970. -N: where -N represents the number of seconds before now, for example: -3600 means 3600 seconds ago, or now - 3600 seconds. For querying purposes, the end_time can be specified as: 0: in which case by default, it means now. N: where N represents the number of seconds since Jan 1, 1970. -N: where -N represents the number of seconds before now, for example: -3600 means 3600 seconds ago, or now - 3600 seconds.  The interval is the suggested sampling interval in seconds.  The default of 0 uses the system default.  The maximum_rows value allows you to limit the returned rows.  Values are started at the start_time and if the number of rows exceeds the value of maximum_rows, then the data is truncated at the maximum_rows value.  A value of 0 implies the default of all rows.

PS> # Allocate a new Query Object
PS> $Query = New-Object -TypeName iControl.SystemStatisticsPerformanceStatisticQuery
PS> $Query.object_name = "CPU"
PS> $Query.start_time = 0
PS> $Query.end_time = 0
PS> $Query.interval = 0
PS> $Query.maximum_rows = 0
PS> # Make method call passing in an array of size one with the specified query
PS> $ReportData = $SystemStats.get_performance_graph_csv_statistics( (,$Query) )
PS> # Look at the contents of the returned data.
PS> $ReportData
object_name    : throughput
start_time     : 1208354160
end_time       : 1208440800
interval       : 240
statistic_data : {116, 105, 109, 101...}

Processing the Data

The statistic_data, much like the ConfigSync's file transfer data, is transferred as a base64 encoded string, which translates to a byte array in .Net.  We will need to convert this byte array into a string and that can be done with the System.Text.ASCIIEncoding class.

PS> # Allocate  a new encoder and turn the byte array into a string
PS> $ASCII = New-Object -TypeName System.Text.ASCIIEncoding
PS> $csvdata = $ASCII.GetString($ReportData[0].statistic_data)
PS> # Look at the resulting dataset
PS> $csvdata
timestamp,"CPU 0","CPU 1"
1208364000,4.3357230000e+00,0.0000000000e+00
1208364240,3.7098920000e+00,0.0000000000e+00
1208364480,3.7187980000e+00,0.0000000000e+00
1208364720,3.3311110000e+00,0.0000000000e+00
1208364960,3.5825310000e+00,0.0000000000e+00
1208365200,3.4826450000e+00,8.3330000000e-03
...

Building a Chart

You will see the returned dataset is in the form of a comma separated value file.  At this point you can take this data and import it into your favorite reporting package.  But, if you want a quick and dirty way to see this visually, you can use PowerShell to control Excel into loading the data and generating a default report.  The following code converts the csv format into a tab separated format, creates an instance of an Excel Application, loads the data, cleans it up, and inserts a default line graph based on the input data.

PS> # Replace commas with tabs in the report data and save to c:\temp\tabdata.txt
PS> $csvdata.Replace(",", "`t") > c:\temp\tabdata.txt
PS> # Allocate an Excel application object and make it visible.
PS> $e = New-Object -comobject "Excel.Application"
PS> $e.visible = $true
PS> # Load the tab delimited data into a workbook and get an instance of the worksheet it was inserted into.
PS> $wb = $e.Workbooks.Open("c:\temp\tabdata.txt")
PS> $ws = $wb.WorkSheets.Item(1)
PS> # Let's remove the first row of timestamps.  Ideally you'll want this to be the 
PS> # horizontal axis and I'll leave it up to you to figure that one out.
PS> $ws.Columns.Item(1).EntireColumn.Select()
PS> $ws.Columns.Item(1).Delete()
PS> # The last row of the data is filled with NaN to indicate the end of the result set. Let's delete that row.
PS> $ws.Rows.Item($ws.UsedRange.Rows.Count).Select()
PS> $ws.Rows.Item($ws.UsedRange.Rows.Count).Delete()
PS> # Select all the data in the worksheet, create a chart, and change it to a line graph.
PS> $ws.UsedRange.Select()
PS> $chart = $e.Charts.Add()
PS> $chart.Type = 4

Conclusion

Now you should have everything you need to get access to the data behind the performance graphs.  There are many ways you could take these examples and I'll leave it to the chart gurus out there to figure out interesting ways to represent the data.  If anyone does find some interesting ways to manipulate this data, please let me know!