Graphing Performance Data with bigsuds
I was asked internally about some performance graphs with python, and while I've done some work with python and iControl over the years, but I've yet to take on graphing. Most of the python scripts I've written output results to the console, so there was no need for fancy things like graphics. I was thinking of a web-based app, but decided to stick with a console app and use the matplotlib 2D plotting libary to generate the graphs. In addition to the normal modules I use for a python script (suds, bigsuds, getpass, argparse) interacting with the BIG-IP, there are few additions:
- base64 - used to decode the iControl stats data returned from BIG-IP
- time - used to reformat the date/time information returned from BIG-IP from epoch to date-time)
- csv - used to read the csv file data to pull out the time field to update it
- numpy - used to convert the string data from the csv file to arrays of float data for use with matplotlib.
- matplotlib - used for generating the graphs and for the graph date formatting of the timestamps. Two modules that are not imported directly in the script but are required nonetheless for matplotlib: dateutil and pyparsing, so you'll need to install those as well before getting started.
Thankfully, Joe wrote on performance graphs for many moons ago in his icontrol 101 #11 article. So the only real work I had to do was python-ize his code, and do my best to one-up his solution!
Grabbing the Data
This section of the main loop consists of importing all the modules we'll need, initializing the BIG-IP, asking the user what graph they'd like to see, and grabbing that data from the BIG-IP via an iControl call.
import bigsuds as pc
import getpass
import argparse
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import numpy as np
import csv, time, base64
parser = argparse.ArgumentParser()
parser.add_argument("-s", "--system", required=True)
parser.add_argument("-u", "--username", required=True)
args = vars(parser.parse_args())
print "%s, enter your " % args['username'],
upass = getpass.getpass()
b = pc.BIGIP(args['system'], args['username'], upass)
stats = b.System.Statistics
graphs = stats.get_performance_graph_list()
graph_obj_name = []
count = 1
print "Graph Options"
for x in graphs:
graph_obj_name.append(x['graph_name'])
print "%d> %s, %s" % (count, x['graph_description'], x['graph_name'])
count +=1
num_select = int(raw_input("Please select the number for the desired graph data: "))
graphstats = stats.get_performance_graph_csv_statistics(
[{'object_name': graph_obj_name[num_select - 1],
'start_time': 0,
'end_time': 0,
'interval': 0,
'maximum_rows': 0}])
In the for loop, I'm just iterating through the data returned by the get_performance_graph_list method and printing it to the console so the user can select the graph they want. Next, I take the user's selection and graph the performance data for that graph with the get_performance_graph_csv_statistics method.
Storing and Manipulating the Data
Now that I have the data from the BIG-IP, I store it in a .csv file just in case the user would like to use it afterward. Before writing it to said .csv however, I need to b64decode the data as show below.
statsFile = "%s_rawStats.csv" % graph_obj_name[num_select-1]
f = open(statsFile, "w")
f.write(base64.b64decode(graphstats[0]['statistic_data']))
f.close()
This results in cvs data that looks like this:
timestamp,"Total Phys Memory","OS Used Memory","TMM Alloc Memory","TMM Used Memory","OS Used Swap" 1387226160, nan, nan, nan, nan, nan 1387226400,4.1607127040e+09,4.1204910763e+09,1.7070817280e+09,6.0823478033e+08,2.4167403520e+08 1387226640,4.1607127040e+09,4.1187021824e+09,1.7070817280e+09,6.0832352860e+08,2.4183606613e+08
After storing the raw data, I re-open that file and read it in line by line, reformatting the timestamp and writing that to a new file:
outputFile = "%s_modStats.csv" % graph_obj_name[num_select-1]
with open(statsFile) as infile, open(outputFile, "w") as outfile:
r = csv.reader(infile)
w = csv.writer(outfile)
w.writerow(next(r))
for row in r:
fl = float(row[0])
row[0] = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(fl))
w.writerow(row)
This results in updated csv data.
timestamp,Total Phys Memory,OS Used Memory,TMM Alloc Memory,TMM Used Memory,OS Used Swap 2013-12-16 14:36:00, nan, nan, nan, nan, nan 2013-12-16 14:40:00,4.1607127040e+09,4.1204910763e+09,1.7070817280e+09,6.0823478033e+08,2.4167403520e+08 2013-12-16 14:44:00,4.1607127040e+09,4.1187021824e+09,1.7070817280e+09,6.0832352860e+08,2.4183606613e+08 I made sure that each graph would end up with its own set of csv files. I'm sure there's a smarter more pythonic way to handle this, but hey, it works.
Calling the Graph Functions
This is the simplest section of code, an if / elif chain! There is no switch statement in python like in Tcl, but there are other solutions available. The if / elif works just fine, so it remains.
if num_select == 1:
graph_memory(outputFile)
elif num_select == 2:
graph_cpu(outputFile)
elif num_select == 3:
graph_active_connections(outputFile)
elif num_select == 4:
graph_new_connections(outputFile)
elif num_select == 5:
graph_throughput(outputFile)
elif num_select == 7:
graph_http_requests(outputFile)
elif num_select == 8:
graph_ramcache_util(outputFile)
elif num_select == 12:
graph_active_ssl(outputFile)
else:
print "graph function not yet defined, please select 1, 2, 3, 4, 5, 7, 8, or 12."
The else is there for the myriad of graphs I have not generated, I'll leave that as an exercise for you!
Generating the Graphs
On my system, collecting the graphs from the get_performance_graph_list method results in 38 different graphs. Many of these graphs have different number of columns of data, so consolidating all these functions into one and working out the data passing and error correction was not an exercise I was willing to take on. So...lots of duplicate code for each graph function, but again, it works. The graphs I have built functions for are memory, cpu, active and new connections, throughput (bits), http requests, ramcache utilization, and active ssl connections. The function is essentially the same for all of them, the changes are related to number of plot lines, labels, etc. The memory graph function is below.
def graph_memory(csvdata):
timestamp, total_mem, os_used, tmm_alloc, tmm_used, swap_used = np.genfromtxt(csvdata, delimiter=',',
skip_header=2, skip_footer=2, unpack=True,
converters= {0: mdates.strpdate2num('%Y-%m-%d %H:%M:%S')})
fig = plt.figure()
plt.plot_date(x=timestamp, y=total_mem, fmt='-')
plt.plot(timestamp, total_mem, 'k-', label="Total Memory")
plt.plot(timestamp, tmm_used, 'r-', label="TMM Used")
plt.plot(timestamp, tmm_alloc, 'c-', label="Tmm Allocated")
plt.plot(timestamp, os_used, 'g-', label="OS Used")
plt.plot(timestamp, swap_used, 'b-', label="OS Used Swap")
plt.legend(title="Context", loc='upper left', shadow=True)
plt.title('Global Memory')
plt.ylabel('GB')
plt.show()
In this function, numpy (np.genfromtxt) is used to pull in the csv data to arrays that matplotlib can use. The rest of the function is just placing and formatting data for the graph.
Running the Script
C:\>python stats1.py -s 192.168.1.1 -u citizen_elah
citizen_elah, enter your Password: Graph Options. Not all options supported at this time, please select only 1, 2, 3, 4, 5, 7, 8, or 12. 1> Memory Used, memory 2> System CPU Usage %, CPU 3> Active Connections, activecons 4> Total New Connections, newcons 5> Throughput(bits), throughput 6> Throughput(packets), throughputpkts 7> HTTP Requests, httprequests 8> RAM Cache Utilization, ramcache 9> Blade Memory Usage, b0memory 10> Active Connections, detailactcons1 11> Active Connections per Blade, detailactcons2 12> Active SSL Connections, detailactcons3 13> New Connections, detailnewcons1 14> New PVA Connections, detailnewcons2 15> New ClientSSL Profile Connections, detailnewcons3 16> New TCP Accepts/Connects, detailnewcons4 17> Client-side Throughput, detailthroughput1 18> Server-side Throughput, detailthroughput2 19> HTTP Compression Rate, detailthroughput3 20> SSL Transactions/Sec, SSLTPSGraph 21> GTM Requests and Resolutions, GTMGraph 22> GTM Requests, GTMrequests 23> GTM Resolutions, GTMresolutions 24> GTM Resolutions Persisted, GTMpersisted 25> GTM Resolutions Returned to DNS, GTMret2dns 26> GTM Requests per DNS type, GTMRequestBreakdownGraph 27> Active Sessions, act_sessions 28> New Sessions, new_sessions 29> Access Requests, access_requests 30> ACL Actions, acl_stats 31> Active Network Access Connections, NAActiveConnections 32> New Network Access Connections, NANewConnections 33> Network Access Throughput, NAthroughput 34> Rewrite Transactions, rewrite_trans 35> Rewrite Transaction Data, rewrite_bytes 36> GTM requests for IPv4 addresses, GTMarequests 37> GTM requests for IPv6 addresses, GTMaaaarequests 38> CPU Usage, blade0cpuset1 Please select the number for the desired graph data: 1
Wrapping Up
This was a fun walk through the matplotlib and the iControl interface to generate some graphs. The script in full is here in the codeshare. Happy coding!
