cancel
Showing results for 
Search instead for 
Did you mean: 
Login & Join the DevCentral Connects Group to watch the Recorded LiveStream (May 12) on Basic iControl Security - show notes included.
Joe_Pruitt
Cirrostratus
Cirrostratus

The BIG-IP stores history of certain types of data for reporting purposes.  The Performance item in the Overview tab shows the report graphs for this performance data.  The management GUI gives you the options to customize the graph interval but it stops short of giving you access to the raw data behind the graphs.  Never fear, the System.Statistics interface contains methods to query the report types and extract the CSV data behind them.  You can even select the start and end times as well as the poll intervals.  This article will discuss the performance graph methods and how you can query the data behind them and build a chart of your own.

Initialization

This article uses PowerShell and the iControl Cmdlets for PowerShell as the client environment for querying the data.  The following setup will be required for the examples contained in this article.

PS> Add-PSSnapIn iControlSnapIn
PS> Initialize-F5.iControl -Hostname bigip_address -Username bigip_user -Password bigip_pass
PS> $SystemStats = (Get-F5.iControl).SystemStatistics

Now that that is taken care of, let's dive right in.  In the System.Statistics interface, there are two methods that get you to the performance graph data.  The first is the get_performance_graph_list() method.  This method takes no parameters and returns a list of structures containing each graph name, title, and description.

PS> $SystemStats.get_performance_graph_list()

graph_name                              graph_title                             graph_description
----------                              -----------                             -----------------
memory                                  Memory Used                             Memory Used
activecons                              Active Connections                      Active Connections
newcons                                 New Connections                         New Connections
throughput                              Throughput                              Throughput
httprequests                            HTTP Requests                           HTTP Requests
ramcache                                RAM Cache Utilization                   RAM Cache Utilization
detailactcons1                          Active Connections                      Active Connections
detailactcons2                          Active PVA Connections                  Active PVA Connections
detailactcons3                          Active SSL Connections                  Active SSL Connections
detailnewcons1                          Total New Connections                   Total New Connections
detailnewcons2                          New PVA Connections                     New PVA Connections
detailnewcons3                          New ClientSSL Profile Connections       New ClientSSL Profile Connections
detailnewcons4                          New Accepts/Connects                    New Accepts/Connects
detailthroughput1                       Client-side Throughput                  Client-side Throughput
detailthroughput2                       Server-side Throughput                  Server-side Throughput
detailthroughput3                       HTTP Compression Rate                   HTTP Compression Rate
SSLTPSGraph                             SSL Transactions/Sec                    SSL Transactions/Sec
GTMGraph                                GTM Performance                         GTM Requests and Resolutions
GTMrequests                             GTM Requests                            GTM Requests
GTMresolutions                          GTM Resolutions                         GTM Resolutions
GTMpersisted                            GTM Resolutions Persisted               GTM Resolutions Persisted
GTMret2dns                              GTM Resolutions Returned to DNS         GTM Resolutions Returned to DNS
detailcpu0                              CPU Utilization                         CPU Usage
detailcpu1                              CPU Utilization                         CPU Usage
CPU                                     CPU Utilization                         CPU Usage
detailtmm0                              TMM Utilization                         TMM Usage
TMM                                     TMM Utilization                         TMM CPU Utilization

Creating a Report

Ok, so you've now got the graph names, it's time to move on to accessing the data.  The method you'll want is the get_performance_graph_csv_statistics() method.  This method takes an array of PerformanceStatisticQuery structures containing the query parameters and returns an array of PerformanceGraphDataCSV structures, one for each input query.  The following code illustrates how to make a simple query.  The object_name corresponds to the graph_name in the get_performance_graph_list() method. 

The start_time and end_time allow you to control what the data range is.  Values of 0 (the default) will return the entire result set.  If the user specifies a start_time, end_time, and interval that does not exactly match the corresponding value used within the database, the database will attempt to use to closest time or interval as requested. The actual values used will be returned to the user on output. For querying purposes, the start_time can be specified as: 0: in which case by default, it means 24 hours ago. N: where N represents the number of seconds since Jan 1, 1970. -N: where -N represents the number of seconds before now, for example: -3600 means 3600 seconds ago, or now - 3600 seconds. For querying purposes, the end_time can be specified as: 0: in which case by default, it means now. N: where N represents the number of seconds since Jan 1, 1970. -N: where -N represents the number of seconds before now, for example: -3600 means 3600 seconds ago, or now - 3600 seconds.  The interval is the suggested sampling interval in seconds.  The default of 0 uses the system default.  The maximum_rows value allows you to limit the returned rows.  Values are started at the start_time and if the number of rows exceeds the value of maximum_rows, then the data is truncated at the maximum_rows value.  A value of 0 implies the default of all rows.

PS> # Allocate a new Query Object
PS> $Query = New-Object -TypeName iControl.SystemStatisticsPerformanceStatisticQuery
PS> $Query.object_name = "CPU"
PS> $Query.start_time = 0
PS> $Query.end_time = 0
PS> $Query.interval = 0
PS> $Query.maximum_rows = 0
PS> # Make method call passing in an array of size one with the specified query
PS> $ReportData = $SystemStats.get_performance_graph_csv_statistics( (,$Query) )
PS> # Look at the contents of the returned data.
PS> $ReportData
object_name    : throughput
start_time     : 1208354160
end_time       : 1208440800
interval       : 240
statistic_data : {116, 105, 109, 101...}

Processing the Data

The statistic_data, much like the ConfigSync's file transfer data, is transferred as a base64 encoded string, which translates to a byte array in .Net.  We will need to convert this byte array into a string and that can be done with the System.Text.ASCIIEncoding class.

PS> # Allocate  a new encoder and turn the byte array into a string
PS> $ASCII = New-Object -TypeName System.Text.ASCIIEncoding
PS> $csvdata = $ASCII.GetString($ReportData[0].statistic_data)
PS> # Look at the resulting dataset
PS> $csvdata
timestamp,"CPU 0","CPU 1"
1208364000,4.3357230000e+00,0.0000000000e+00
1208364240,3.7098920000e+00,0.0000000000e+00
1208364480,3.7187980000e+00,0.0000000000e+00
1208364720,3.3311110000e+00,0.0000000000e+00
1208364960,3.5825310000e+00,0.0000000000e+00
1208365200,3.4826450000e+00,8.3330000000e-03
...

Building a Chart

You will see the returned dataset is in the form of a comma separated value file.  At this point you can take this data and import it into your favorite reporting package.  But, if you want a quick and dirty way to see this visually, you can use PowerShell to control Excel into loading the data and generating a default report.  The following code converts the csv format into a tab separated format, creates an instance of an Excel Application, loads the data, cleans it up, and inserts a default line graph based on the input data.

PS> # Replace commas with tabs in the report data and save to c:\temp\tabdata.txt
PS> $csvdata.Replace(",", "`t") > c:\temp\tabdata.txt
PS> # Allocate an Excel application object and make it visible.
PS> $e = New-Object -comobject "Excel.Application"
PS> $e.visible = $true
PS> # Load the tab delimited data into a workbook and get an instance of the worksheet it was inserted into.
PS> $wb = $e.Workbooks.Open("c:\temp\tabdata.txt")
PS> $ws = $wb.WorkSheets.Item(1)
PS> # Let's remove the first row of timestamps.  Ideally you'll want this to be the 
PS> # horizontal axis and I'll leave it up to you to figure that one out.
PS> $ws.Columns.Item(1).EntireColumn.Select()
PS> $ws.Columns.Item(1).Delete()
PS> # The last row of the data is filled with NaN to indicate the end of the result set. Let's delete that row.
PS> $ws.Rows.Item($ws.UsedRange.Rows.Count).Select()
PS> $ws.Rows.Item($ws.UsedRange.Rows.Count).Delete()
PS> # Select all the data in the worksheet, create a chart, and change it to a line graph.
PS> $ws.UsedRange.Select()
PS> $chart = $e.Charts.Add()
PS> $chart.Type = 4
0151T000003d7dIQAQ.jpg

Conclusion

Now you should have everything you need to get access to the data behind the performance graphs.  There are many ways you could take these examples and I'll leave it to the chart gurus out there to figure out interesting ways to represent the data.  If anyone does find some interesting ways to manipulate this data, please let me know!

Get the Flash Player to see this player.
Comments
Joe_Hsy_45207
Nimbostratus
Nimbostratus
Hi Joe,

 

 

Great post as always! However, do you know if there is version dependency?

 

 

We were able to get the powershell script example (and our own C implementation) working flawlessly on BIG-IP 9.2.5 Build 5.1, but it always hangs the apache (thus all iControl) on BIG-IP 9.4.0 Build 517.5. I looked on support and found no bug report on that specific 9.4.0 build.

 

 

Since we are building a product that needs to work across versions and it would not do to hang icontrol, this is of great concern to us. Any pointer would be greatly appreciated!

 

 

Thanks!

 

 

//Joe
Joe_Hsy_45207
Nimbostratus
Nimbostratus
Just as a follow-up, everytime we call get_performance_graph_csv_statistics, it would generate a statsd reset warning on the ssh console as follows:

 

 

Oct 16 15:56:26 biggerip LOGIN: Re-starting statsd

 

 

And looking in /var/log/ltm, the following entries show the same:

 

 

Oct 16 15:49:11 biggerip iControlPortal.cgi[6369]: System:+++++++++++++++++++++++++++++++++++++

 

Oct 16 15:49:11 biggerip iControlPortal.cgi[6369]: System:Statistics::get_performance_graph_csv_statistics ( ) called by user "admin"

 

Oct 16 15:49:11 biggerip iControlPortal.cgi[6369]: System: [0] Object: memory

 

Oct 16 15:49:11 biggerip iControlPortal.cgi[6369]: System: Start: 0

 

Oct 16 15:49:11 biggerip iControlPortal.cgi[6369]: System: End: 0

 

Oct 16 15:49:11 biggerip iControlPortal.cgi[6369]: System: Sample interval: 0

 

Oct 16 15:49:11 biggerip iControlPortal.cgi[6369]: System: Max. rows: 0

 

Oct 16 15:49:11 biggerip iControlPortal.cgi[6369]: System:-------------------------------------

 

Oct 16 15:49:11 biggerip iControlPortal.cgi[6369]: Portal:User/Partition map:

 

Oct 16 15:49:11 biggerip iControlPortal.cgi[6369]: Portal: User: admin, Partition: Common

 

Oct 16 15:49:11 biggerip mcpd[2090]: 01070406:5: Removed publication with publisher id STATSD

 

Oct 16 15:49:12 biggerip mcpd[2090]: 01070404:5: Add a new Publication for publisherID STATSD and filterType 0x8000000

 

Oct 16 15:49:36 biggerip mcpd[2090]: 01070418:5: connection 0x9289ca0 (user admin) was closed with active requests

 

Oct 16 15:49:38 biggerip iControlPortal.cgi[6728]: icontrol_portal:iControl SOAP Portal starting ...

 

 

Let us know if we need to open a case with support. Thanks again,

 

 

//Joe

 

 

 

Joe_Hsy_45207
Nimbostratus
Nimbostratus
Just an update - we upgraded to 4.5 and the issue no longer exists. I guess this means there is a bug in 9.4.0 build 517.5.
George_32256
Nimbostratus
Nimbostratus
How do you do this with pycontrol?
Macaron
Altostratus
Altostratus

Hi I ran this script and it worked really well. I have a question, right now the result on CSV are after 30 seconds gap. Is there a way to reduce the gap to 26 seconds ?

 

Right now if I collect statistic manually, I get result of excel after 26 seconds gap, i want get the same result via this script

 

Thanks

 

Version history
Last update:
‎17-Apr-2008 11:44
Updated by:
Contributors