Python module to post and retrieve IControl Rest JSON objects for AVR statistics
Problem this snippet solves:
This module simplifies making Python dictionary objects that are converted to IControl rest AVR JSON objects. It also handles making AVR requests and retrieving results as well allowing multiple AVR requests to be queued, posted and retrieved. It also has some basis type checking for the elements of a AVR request. This module requires Bigip 12.1 on the target that statistics are retrieved.
How to use this snippet:
The main class is rest_avr.avr_req. It is a dictionary class that maps directly to an IControl Rest AVR JSON request as translated by json.dumps. Each dictionary element is an object derived from a customer class for each part of the request. The element classes have add() and clear() functions. if the element class only allows one entry the add() function will replace the existing entry, otherwise it will append the entry to the request element. The rest_avr.avr_req class also has functions to populate the HTTP host and authentication values for the target system.
rest_avr.avr_req.post_and_responserest_avr.avr_req.add_to_queue()rest_avr.avr_req.post_and_response_queue()The following code sample constructs, posts and returns results for an AVR statistics request for specific DNS records and a specificrecord type, then queues multiple quests and posts and returns results.
#!/usr/bin/python import json import sys import time import rest_avr #print rest_avr.ShowAVRJsonApi #Populate the url avr_dns_req=rest_avr.avr_req() avr_dns_req.auth('admin','admin') avr_dns_req.url_base('10.10.2.113','dns') #Populate the json object avr_dns_req['analyticsModule'].add('dns') avr_dns_req['reportFeatures'].add('time-aggregated') avr_dns_req['entityFilters'].add('domain-name', 'OPERATOR_TYPE_EQUAL', ['test2.test1.com','test1.test1.com']) avr_dns_req['entityFilters'].add('query-type', 'OPERATOR_TYPE_EQUAL', ['a']) avr_dns_req['viewMetrics'].add('packets') avr_dns_req['viewDimensions'].add('domain-name') avr_dns_req['metricFilters'].add('packets', 'OPERATOR_TYPE_GREATER_THAN', 0) avr_dns_req['sortByMetrics'].add('packets', 'ascending') avr_dns_req['pagination'].add(20, 0) avr_dns_req['timeRange'].add(1461778251000000, None) #Post and retrieve results. result_py=avr_dns_req.post_and_response() if result_py != None: print ('\n' + result_py['results']['timeAggregated'][0]['dimensions'][0]['value'] + " " + result_py['results']['timeAggregated'][0]['metricValues'][0]['value'] + '\n') else: print result_py.error_layer print result_py.error_code print result_py.error_text # Now add multiple requests to a queue avr_dns_req.add_to_queue() avr_dns_req['entityFilters'].clear() avr_dns_req['entityFilters'].add('query-type', 'OPERATOR_TYPE_EQUAL', ['aaaa']) avr_dns_req.add_to_queue() #post and retrieve queued results result_py_q=avr_dns_req.post_and_response_queue() for result_py in result_py_q: if result_py != None: print ('\n' + result_py['results']['timeAggregated'][0]['dimensions'][0]['value'] + " " + result_py['results']['timeAggregated'][0]['metricValues'][0]['value'] + '\n') else: print result_py.error_layer print result_py.error_code print result_py.error_text
Code :
"""
rest_avr provides a python interface to Bigip AVR statistics using the REST API. The main Python rest_avr.avr_req
object is a Python dictionary that maps to a JSON object that can be processed with the json.dumps() function
An IControl Rest AVR JSON request and response can be initiated with avr_req.post_and_response
The simple description of the API can is available at avr_req.ShowJsonApi()
Each of these modules has a method to add single or multiple elements as appropriate to the specific module.
Once these elements are are populated a RestAPI request can be made with results returned as a python
representation.
avr_req.auth(user, passw)
avr_req.url_base(host, module)
avr_req['analyticsModule'].add(module)
avr_req['analyticsModule'].clear()
avr_req['reportFeatures'].add(metric_name, predicate, value)
avr_req['reportFeatures'].clear()
avr_req['entityFilters'].add(dimension_name, predicate, values)
avr_req['entityFilters'].clear()
avr_req['viewMetrics'].add(metric_name)
avr_req['viewMetrics'].clear()
avr_req['viewDimensions'].add(metric_name, order)
avr_req['viewDimensions'].clear()
avr_req['metricFilters'].add(metric_name, predicate, valu)
avr_req['metricFilters'].clear()
avr_req['sortByMetrics'].add(metric_name, orde)
avr_req['sortByMetrics'].clear()
avr_req['pagination'].add(num_results, skip_result)
avr_req['pagination'].clear()
avr_req['timeRange'].add(t_from, t_to)
avr_req['timeRange'].clear()
After a request in constructed a REST API call is initiated with initiated with:
avr_req.post_and_response()
The response is a python dictionary data structure of the results as processed by json.loads
"""
from copy import deepcopy
import requests
import json
import sys
import time
import warnings
__author__ = 'Mark Lloyd'
__version__ = '1.0'
# 05/24/2016
import json
import requests
import time
class BadDictElement(Exception):
def __init__(self, key, value, expl):
Exception.__init__(self, '{0} {1} {2} '.format(key, value, expl))
class BadTime(Exception):
def __init__(self, variable, value):
Exception.__init__(self, '{0} {1} should be 16 char decimal in microseconds '.format('a', 'b'))
class RequestFailure(Exception):
def __init__(self, key, value):
Exception.__init__(self, '{0} {1} '.format(key, value))
class analyticsModule(str):
"""
This class is tied to the structure of the parent class.
parent() get's the parent object so we can make the string pseudo-mutable.
accessed from within an avr request
['analyticsModule'].add(module)
Adds a single string to analyticsModule element . If one exists it is replaced.
['analyticsModule'].clear()
Send a null value to the analyticsModule element.
See rest_avr.ShowAVRJsonApi for more details
"""
def parent(self, parent):
self.parent = parent
def add(self, module):
"""
avr_req.['analyticsModule'].add(module)
Adds a single string to analyticsModule element . If one already exists it is replaced.
This should be the same as the module string in avr_req.url_base.
"""
self.parent['analyticsModule'] = analyticsModule(module)
self.parent['analyticsModule'].parent = self.parent
def clear(self):
"""
avr_req.['analyticsModule'].add(module)
replaces the analyticsModule mddule with a null string
"""
self.parent['analyticsModule'] = analyticsModule('')
self.parent['analyticsModule'].parent = self.parent
class metricFilters(list):
"""
avr_req.['metricFilters'].add(metric_name, predicate, value)
metric name is a string, value is an integer
Valid predicates strings are ['OPERATOR_TYPE_EQUAL', 'OPERATOR_TYPE_NOT_EQUAL', 'OPERATOR_TYPE_GREATER_THAN',
OPERATOR_TYPE_LOWER_THAN','OPERATOR_TYPE_GREATER_THAN_OR_EQUAL', 'OPERATOR_TYPE_LOWER_THAN_OR_EQUAL'])
avr_req['metricFilters'].clear()
Clears metricFilters elements
See rest_avr.ShowAVRJsonApi for more details.
"""
def __init__(self):
self.append([])
self.valid_metric_predicate = (
['OPERATOR_TYPE_EQUAL', 'OPERATOR_TYPE_NOT_EQUAL', 'OPERATOR_TYPE_GREATER_THAN', 'OPERATOR_TYPE_LOWER_THAN',
'OPERATOR_TYPE_GREATER_THAN_OR_EQUAL', 'OPERATOR_TYPE_LOWER_THAN_OR_EQUAL'])
def add(self, metric_name, predicate, value):
"""
avr_req.['metricFilters'].add(metric_name, predicate, value)
metric name is a string, value is an integer
Valid predicates strings are ['OPERATOR_TYPE_EQUAL', 'OPERATOR_TYPE_NOT_EQUAL', 'OPERATOR_TYPE_GREATER_THAN',
OPERATOR_TYPE_LOWER_THAN','OPERATOR_TYPE_GREATER_THAN_OR_EQUAL', 'OPERATOR_TYPE_LOWER_THAN_OR_EQUAL']
"""
if type(value) is not int:
raise BadDictElement(metric_name, value, 'value should be integer')
if predicate in self.valid_metric_predicate:
# first check if it is already there
for metric in self[0]:
if metric['metricName'] == metric_name:
metric['predicate'] = predicate
metric['value'] = value
return 0
# if it is not there then just add it.
self[0].append({'metricName': metric_name, 'predicate': predicate, 'value': value})
else:
raise BadDictElement(metric_name, predicate, 'invalid predicate')
def clear(self):
"""
avr_req['metricFilters'].clear()
Clears metricFilters elements
"""
del self[0][:]
class entityFilters(list):
"""
avr_req.['entityFilters'].add(dimension_name, predicate, values):
All values are strings
valid predicate is 'OPERATOR_TYPE_EQUAL'
['entityFilters'].clear()
Clears the entityFilters element
See rest_avr.ShowJsonApi for more details
"""
def __init__(self):
self.append([])
def add(self, dimension_name, predicate, values):
"""
avr_req.['entityFilters'].add(dimension_name, predicate, values):
All values are strings
valid predicate is 'OPERATOR_TYPE_EQUAL'
"""
if predicate is 'OPERATOR_TYPE_EQUAL':
# then loop throuth to see if the dimenson name already exists, if so replace
for entity in self[0]:
if entity['dimensionName'] == dimension_name:
entity['predicate'] = predicate
entity['values'] = values
return 0
# if it is not there then just add it.
self[0].append({'dimensionName': dimension_name, 'predicate': predicate, 'values': values})
else:
raise BadDictElement(dimension_name, predicate, 'predicate must be OPERATOR_TYPE_EQUAL')
def clear(self):
"""
['entityFilters'].clear()
Clears the entityFilters element
"""
del self[0][:]
class reportFeatures(list):
"""
avr_req.['reportFeatures'].add( feature)
adds report feature string. Multiple features are permitted.
['reportFeatures'].clear()
Clears the analyticsModule element.
See rest_avr.ShowAVRJsonApi for more details.
"""
def add(self, feature):
"""
avr_req.['reportFeatures'].add( feature)
adds report feature string. Multiple features are permitted
."""
if feature not in self:
self.append(feature)
def clear(self):
"""
['reportFeatures'].clear()
Clears the entityFilters element
"""
del self[:]
class sortByMetrics(list):
"""
avr_req.['sortByMetrics'].add(metric_name, order)
valid order names are 'ascending' and 'descending'
sortByMetrics is optional in an AVR request.
avr_req['sortByMetrics'].clear()
Clears the sortByMetrics element.
See rest_avr.ShowAVRJsonApi for more details.
"""
def __init__(self):
self.metric_list = []
def add(self, metric_name, order):
if metric_name not in self.metric_list:
self.append({'metricName': metric_name, 'order': order})
self.metric_list.append(metric_name)
def clear(self):
"""
['sortByMetrics'].clear()
Clears the sortByMetrics element
"""
del self[:]
del self.metric_list[:]
class viewDimensions(list):
"""
avr_req.['viewDimensions'].add(dimension_name):
adds view dimension, only one dimension is allowed
add will replace element if it already exists
avr_req['viewDimensions'].clear()
Clears the viewDimensions element.
See rest_avr.ShowAVRJsonApi for more details.
"""
def __init__(self):
self.append([])
self[0] = {}
def add(self, dimension_name):
"""
avr_req.['viewDimensions'].add(dimension_name):
adds view dimension string, only one dimension is allowed
add will replace element if it already exists
"""
self[0]['dimensionName'] = dimension_name
def clear(self, dimension_name):
"""
['viewDimensions'].clear()
Clears the viewDimensions element
"""
del self[0][:]
class viewMetrics(list):
"""
avr_req.['viewMetrics'].add(metric_name):
appends metric_name string to list. The specification
allows multiple view metric elements
avr_req['viewMetrics'].clear()
Clears the viewMetrics elements
See rest_avr.ShowAVRJsonApi for more details.
"""
def __init__(self):
self.metric_list = []
def add(self, metric_name):
"""
avr_req.['viewMetrics'].add(metric_name):
appends metric_name string to list. The specification
allows multiple viewMetric elements
"""
if metric_name not in self.metric_list:
self.append({'metricName': metric_name})
self.metric_list.append(metric_name)
def clear(self):
"""
['viewMetrics'].clear()
Clears the viewMetrics elements
"""
del self[:]
del self.metric_list[:]
class timeRange(dict):
"""
avr_req.['timeRange'].add( t_from, t_to)
both values are 16 digit numeric value in microseconds
of unix/linux time. t_to is optional and can be replace by None
timeRange is an optional.
avr_req['timeRange'].clear()
Clears the timeRange elements
See rest_avr.ShowAVRJsonApi for more details.
"""
def add(self, t_from, t_to):
"""
avr_req.['timeRange'].add( t_from, t_to)
both values are 16 digit numeric value in microseconds
of unix/linux time. t_to is optional and can be replace by None
timeRange is optional.
"""
if type(t_from) is long and len(str(t_from)) == 16:
self['from'] = t_from
else:
raise BadTime(t_from + " is 16 digit numeric value in microseconds")
if t_to != '' and t_to != 0 and t_to != None:
if type(t_to) is long and len(str(t_from)) == 16:
self['to'] = t_to
else:
raise BadTime(t_to + " is 16 digit numeric value in microseconds")
else:
if 'to' in self.keys():
del self['to']
def clear(self):
"""
['timeRange'].clear()
Clears the timeRange element
"""
del self[:]
class pagination(dict):
"""
avr_req.['pagination'].add(num_results, skip_results)
both are integer values.
avr_req['pagination'].clear()
Clears the pagination elements
See rest_avr.ShowAVRJsonApi for more details.
"""
def add(self, num_results, skip_results):
"""
avr_req.['pagination'].add(num_results, skip_results)
both arguments are integers.
"""
if type(num_results) is int:
self['numberOfResults'] = num_results
else:
raise BadDictElement('number of Results ', num_results, 'must be integer')
if type(skip_results) is int:
self['skipResults'] = skip_results
else:
raise BadDictElement('skipResults ', skip_results, 'must be integer')
def clear(self):
"""
['pagination'].clear()
Clears the pagination element
"""
del self[:]
class avr_resp(dict):
"""
python response error is applicable.
"""
def __init__(self):
self.error_layer = None
self.error_code = None
self.error_text = None
class avr_req(dict):
"""
The main class for rest_avr.
avr_req contains a dictionary that maps to the elements of a Icontrol REST AVR request along with capability of
posting that request and receiving a response.
The dictionary values are object instances of python classes that correspond to the the JSON values of the
object's name/value pair.
Each value has two public methods:
avr_req.['objectName']add(): adds an element to the appropriate object with type checking.
If an element allows more then one instance the add function will append the element
If an element allows only one instance the add function will replace the element
avr_req.['objectName'].clear()r: clears all elements in the object.
printing rest_avr.ShowAVRJsonApi provides documentation for the AVR JASON elements. Further documentation is
available on devcentral.f5.com
To post an AVR Rest request there are two functions to populate the HTTP/HTTPS request.
avr_req.auth(user, passw):
provides the username and password
avr_req.url_base(host, module)
provides the host and the bigip module AVR queries to construct the URL to make the request.
Then to post the request and return results in a python representation of the JSON response.
avr_req.post_and_response()
"""
def __init__(self):
self['analyticsModule'] = analyticsModule()
self['analyticsModule'].parent = self
self['pagination'] = pagination()
self['metricFilters'] = metricFilters()
self['entityFilters'] = entityFilters()
self['reportFeatures'] = reportFeatures()
self['sortByMetrics'] = sortByMetrics()
self['viewDimensions'] = viewDimensions()
self['viewMetrics'] = viewMetrics()
self['timeRange'] = timeRange()
self.avr_session = requests.session()
self.avr_session.verify = False
self.avr_session.headers.update({'Content-Type': 'application/json'})
# for multiple queued request handling.
self.req_queue = []
self.generate_id = None
self.done = None
self.result = None
self.num_requests = 0
self.res_queue = []
def post_and_response(self):
"""
returns a python representation of the json response to the request.
failure returns array ['ERROR','component',error]
"""
warnings.filterwarnings("ignore")
self.generate_request = self.avr_session.post(self.req_url_base + "/generate-report/",
data=json.dumps(self))
self.generate_request_py = json.loads(self.generate_request.text)
self.result_guid = self.generate_request_py['id']
self.results_status_url = self.req_url_base + "/generate-report/" + self.result_guid + "/?$select=status,reportResultsLink"
self.results_url = self.req_url_base + "/report-results/" + self.result_guid
self.sleeptime = .5
for i in range(5):
time.sleep(self.sleeptime)
self.sleeptime *= 2 # double backoff period each time.
self.status_results_json = self.avr_session.get(self.results_status_url)
self.status_results = json.loads(self.status_results_json.text)
if self.status_results['status'] == 'FAILED':
self.result = avr_resp()
self.result_error_layer = 'REST'
self.result_error_code = self.status_results['status']
self.result.error_text = self.status_results
if self.status_results['status'] == 'FINISHED':
self.raw_results_url = self.status_results['reportResultsLink']
self.results_url = self.raw_results_url.replace('localhost', self.host_name)
self.results = self.avr_session.get(self.results_url)
if self.results.status_code == 200:
self.result = avr_resp()
self.result.update(json.loads(self.results.text))
return self.result
else:
self.result = avr_resp()
self.result.error_layer = 'HTTP'
self.result.error_code = self.results.status_code
self.result.error_text = self.results
return self.result
else:
continue
self.result = avr_resp()
self.result.error_layer = 'REST_AVR'
self.result.error_code = '408'
self.result.error_text = 'TIMEOUT'
def auth(self, user, passw):
"""
avr_req.auth(user, passw):
username and password
"""
self.avr_session.auth = (user, passw)
def url_base(self, host, module):
"""
avr_req.url_base(host, module)
host and bigip module AVR queries to construct the URL to make the request.
"""
self.host_name = host
self.req_url_base = 'https://%s/mgmt/tm/analytics/%s' % (host, module)
self.module_py = {'analyticsModule': module}
def add_to_queue(self):
"adds request as currently constructed to queue"
self.req_queue.append(deepcopy(self))
def clear_queue(self):
""""
clears request queue
"""
del self.req_queue[:]
def post_and_response_queue(self):
"""
posts and sends response to from queue of requests.
"""
warnings.filterwarnings("ignore")
for req in self.req_queue:
req.generate_request = req.avr_session.post(req.req_url_base + "/generate-report/",
data=json.dumps(req))
req.generate_request_py = json.loads(req.generate_request.text)
req.generate_id = (req.generate_request_py['id'])
req.results_status_url = self.req_url_base + "/generate-report/" + req.generate_id + "/?$select=status,reportResultsLink"
self.sleeptime = .5
self.num_requests = len(self.req_queue)
for i in range(5):
for req in self.req_queue:
if req.done is None:
time.sleep(self.sleeptime)
self.sleeptime *= 2 # double backoff period each time.
req.status_results_json = req.avr_session.get(req.results_status_url)
req.status_results = json.loads(req.status_results_json.text)
if req.status_results['status'] == 'FAILED':
req.result = avr_resp()
req.result_error['layer'] = 'REST'
req.result_error['error'] = req.status_results['status']
req.result_error['text'] = req.status_results
if req.status_results['status'] == 'FINISHED':
req.raw_results_url = req.status_results['reportResultsLink']
req.results_url = req.raw_results_url.replace('localhost', self.host_name)
req.results = self.avr_session.get(req.results_url)
if req.results.status_code == 200:
req.result = avr_resp()
req.result.update(json.loads(req.results.text))
req.done = True
self.res_queue.append(req.result)
self.num_requests -= 1
else:
req.result = avr_resp()
req.result_error.layer = 'HTTP'
req.result_error.code = req.results.status_code
req.result_error.text = req.results
self.res_queue.append(req.result)
if i == 5:
if req.result == False:
req.result = avr_resp()
req.result.error_layer = 'REST_AVR'
req.result.error_error = '408'
req.result.error_text = 'TIMEOUT'
if self.num_requests == 0:
break
return self.res_queue
ShowAVRJsonApi = """
reportFeatures
--------------
Specifies the kind of information that appears in a
response from AVR. You may specify one or more of the
following values:
existing-entities
time-aggregated
time-series
entities-count
viewDimensions
--------------
Specifies the dimensions for which to calculate a report,
such as:
{"dimensionName": "domain-name"}
You may only specify a single dimension. You may omit
this field in a report generation request.
viewMetrics
-----------
Specifies the list of metrics by which to sort results, such
as:
{ "metricName": "average-tps" },
{ "metricName": "transactions" }
If you specify either time-aggregated or
time-series features, you must specify one metric in
a report generation request.
sortByMetrics
--------------
Specifies the list of metrics to sort by, such as:
[{ metricName: "average-tps", order:"descending" } ]
Valid values are ascending and descending. Sorting
only applies to the time-aggregated feature. You do
not need to specify this field in a report generation request.
timeRange
---------
Specifies the time range, in microseconds, for which to
calculate a report, such as:
{"from": 1410420888000000, "to": 1410424488000000 }
You do not need to specify this field in a report generation
request.
entityFilters
=============
Specifies the entities and values for which to calculate a
report. You can specify a single entity with a second level
of dimension filters that describe an aspect of the entity.
If you specify multiple entity types, the results include
only the entities that match all of the criteria. You do not
need to specify this field in a report generation request.
The following snippet contains two entities with
corresponding values:
[[{
"dimensionName" : "virtual",
"predicate": "OPERATOR_TYPE_EQUAL",
"values : ["phpAuction_VS_1"] },
{
"dimensionName : "response-code",
"predicate": "OPERATOR_TYPE_EQUAL",
"values" : ["200"] }
]]
metricFilters
-------------
Specifies the metric filters for which to calculate a report,
such as:
[{ "metricName": "transactions",
"predicate" :
metricFilters
"OPERATOR_TYPE_GREATER_THAN" "value":
100 }]
You do not need to specify this field in a report generation
request. For the existing-entities feature, AVR
supports the OPERATOR_TYPE_LIKE predicate. AVR also
supports the following predicates:
OPERATOR_TYPE_EQUAL
OPERATOR_TYPE_NOT_EQUAL
OPERATOR_TYPE_GREATER_THAN
OPERATOR_TYPE_LOWER_THAN
OPERATOR_TYPE_GREATER_THAN_OR_EQUAL
OPERATOR_TYPE_LOWER_THAN_OR_EQUAL
pagination
----------
Specifies the number of results to return, and the number
of results to skip, such as:
{ numberOfResults : 10, skipResults : 10}
To see the second set of ten results, use the example
shown here. AVR does not implement the OData query
parameters top or skip. In order to see a specific set of
results, you must set the number of results to return and
then determine how many results to skip. You do not need
to specify this field in a report generation request.
"""Tested this on version:
12.0Updated Jun 06, 2023
Version 2.0
Mark_Lloyd
Employee
Employee
ShamatulskiiMist
Hi Mark_Lloyd,
Could you please help me find the rest_avr library? Also, could you confirm whether the script is working correctly? I would really appreciate your assistance with this!
Thank you!