Python module to post and retrieve IControl Rest JSON objects for AVR statistics
Problem this snippet solves: This module simplifies making Python dictionary objects that are converted to IControl rest AVR JSON objects. It also handles making AVR requests and retrieving results as well allowing multiple AVR requests to be queued, posted and retrieved. It also has some basis type checking for the elements of a AVR request. This module requires Bigip 12.1 on the target that statistics are retrieved. How to use this snippet: The main class is rest_avr.avr_req. It is a dictionary class that maps directly to an IControl Rest AVR JSON request as translated by json.dumps. Each dictionary element is an object derived from a customer class for each part of the request. The element classes have add() and clear() functions. if the element class only allows one entry the add() function will replace the existing entry, otherwise it will append the entry to the request element. The rest_avr.avr_req class also has functions to populate the HTTP host and authentication values for the target system. rest_avr.avr_req.post_and_response returns the Python representation of the JSON result of the query. rest_avr.avr_req.add_to_queue() adds the currently constructed request to a queue of requests to post. rest_avr.avr_req.post_and_response_queue() returns a python list of results of queued queries. The following code sample constructs, posts and returns results for an AVR statistics request for specific DNS records and a specificrecord type, then queues multiple quests and posts and returns results. #!/usr/bin/python import json import sys import time import rest_avr #print rest_avr.ShowAVRJsonApi #Populate the url avr_dns_req=rest_avr.avr_req() avr_dns_req.auth('admin','admin') avr_dns_req.url_base('10.10.2.113','dns') #Populate the json object avr_dns_req['analyticsModule'].add('dns') avr_dns_req['reportFeatures'].add('time-aggregated') avr_dns_req['entityFilters'].add('domain-name', 'OPERATOR_TYPE_EQUAL', ['test2.test1.com','test1.test1.com']) avr_dns_req['entityFilters'].add('query-type', 'OPERATOR_TYPE_EQUAL', ['a']) avr_dns_req['viewMetrics'].add('packets') avr_dns_req['viewDimensions'].add('domain-name') avr_dns_req['metricFilters'].add('packets', 'OPERATOR_TYPE_GREATER_THAN', 0) avr_dns_req['sortByMetrics'].add('packets', 'ascending') avr_dns_req['pagination'].add(20, 0) avr_dns_req['timeRange'].add(1461778251000000, None) #Post and retrieve results. result_py=avr_dns_req.post_and_response() if result_py != None: print ('\n' + result_py['results']['timeAggregated'][0]['dimensions'][0]['value'] + " " + result_py['results']['timeAggregated'][0]['metricValues'][0]['value'] + '\n') else: print result_py.error_layer print result_py.error_code print result_py.error_text # Now add multiple requests to a queue avr_dns_req.add_to_queue() avr_dns_req['entityFilters'].clear() avr_dns_req['entityFilters'].add('query-type', 'OPERATOR_TYPE_EQUAL', ['aaaa']) avr_dns_req.add_to_queue() #post and retrieve queued results result_py_q=avr_dns_req.post_and_response_queue() for result_py in result_py_q: if result_py != None: print ('\n' + result_py['results']['timeAggregated'][0]['dimensions'][0]['value'] + " " + result_py['results']['timeAggregated'][0]['metricValues'][0]['value'] + '\n') else: print result_py.error_layer print result_py.error_code print result_py.error_text Code : """ rest_avr provides a python interface to Bigip AVR statistics using the REST API. The main Python rest_avr.avr_req object is a Python dictionary that maps to a JSON object that can be processed with the json.dumps() function An IControl Rest AVR JSON request and response can be initiated with avr_req.post_and_response The simple description of the API can is available at avr_req.ShowJsonApi() Each of these modules has a method to add single or multiple elements as appropriate to the specific module. Once these elements are are populated a RestAPI request can be made with results returned as a python representation. avr_req.auth(user, passw) avr_req.url_base(host, module) avr_req['analyticsModule'].add(module) avr_req['analyticsModule'].clear() avr_req['reportFeatures'].add(metric_name, predicate, value) avr_req['reportFeatures'].clear() avr_req['entityFilters'].add(dimension_name, predicate, values) avr_req['entityFilters'].clear() avr_req['viewMetrics'].add(metric_name) avr_req['viewMetrics'].clear() avr_req['viewDimensions'].add(metric_name, order) avr_req['viewDimensions'].clear() avr_req['metricFilters'].add(metric_name, predicate, valu) avr_req['metricFilters'].clear() avr_req['sortByMetrics'].add(metric_name, orde) avr_req['sortByMetrics'].clear() avr_req['pagination'].add(num_results, skip_result) avr_req['pagination'].clear() avr_req['timeRange'].add(t_from, t_to) avr_req['timeRange'].clear() After a request in constructed a REST API call is initiated with initiated with: avr_req.post_and_response() The response is a python dictionary data structure of the results as processed by json.loads """ from copy import deepcopy import requests import json import sys import time import warnings __author__ = 'Mark Lloyd' __version__ = '1.0' # 05/24/2016 import json import requests import time class BadDictElement(Exception): def __init__(self, key, value, expl): Exception.__init__(self, '{0} {1} {2} '.format(key, value, expl)) class BadTime(Exception): def __init__(self, variable, value): Exception.__init__(self, '{0} {1} should be 16 char decimal in microseconds '.format('a', 'b')) class RequestFailure(Exception): def __init__(self, key, value): Exception.__init__(self, '{0} {1} '.format(key, value)) class analyticsModule(str): """ This class is tied to the structure of the parent class. parent() get's the parent object so we can make the string pseudo-mutable. accessed from within an avr request ['analyticsModule'].add(module) Adds a single string to analyticsModule element . If one exists it is replaced. ['analyticsModule'].clear() Send a null value to the analyticsModule element. See rest_avr.ShowAVRJsonApi for more details """ def parent(self, parent): self.parent = parent def add(self, module): """ avr_req.['analyticsModule'].add(module) Adds a single string to analyticsModule element . If one already exists it is replaced. This should be the same as the module string in avr_req.url_base. """ self.parent['analyticsModule'] = analyticsModule(module) self.parent['analyticsModule'].parent = self.parent def clear(self): """ avr_req.['analyticsModule'].add(module) replaces the analyticsModule mddule with a null string """ self.parent['analyticsModule'] = analyticsModule('') self.parent['analyticsModule'].parent = self.parent class metricFilters(list): """ avr_req.['metricFilters'].add(metric_name, predicate, value) metric name is a string, value is an integer Valid predicates strings are ['OPERATOR_TYPE_EQUAL', 'OPERATOR_TYPE_NOT_EQUAL', 'OPERATOR_TYPE_GREATER_THAN', OPERATOR_TYPE_LOWER_THAN','OPERATOR_TYPE_GREATER_THAN_OR_EQUAL', 'OPERATOR_TYPE_LOWER_THAN_OR_EQUAL']) avr_req['metricFilters'].clear() Clears metricFilters elements See rest_avr.ShowAVRJsonApi for more details. """ def __init__(self): self.append([]) self.valid_metric_predicate = ( ['OPERATOR_TYPE_EQUAL', 'OPERATOR_TYPE_NOT_EQUAL', 'OPERATOR_TYPE_GREATER_THAN', 'OPERATOR_TYPE_LOWER_THAN', 'OPERATOR_TYPE_GREATER_THAN_OR_EQUAL', 'OPERATOR_TYPE_LOWER_THAN_OR_EQUAL']) def add(self, metric_name, predicate, value): """ avr_req.['metricFilters'].add(metric_name, predicate, value) metric name is a string, value is an integer Valid predicates strings are ['OPERATOR_TYPE_EQUAL', 'OPERATOR_TYPE_NOT_EQUAL', 'OPERATOR_TYPE_GREATER_THAN', OPERATOR_TYPE_LOWER_THAN','OPERATOR_TYPE_GREATER_THAN_OR_EQUAL', 'OPERATOR_TYPE_LOWER_THAN_OR_EQUAL'] """ if type(value) is not int: raise BadDictElement(metric_name, value, 'value should be integer') if predicate in self.valid_metric_predicate: # first check if it is already there for metric in self[0]: if metric['metricName'] == metric_name: metric['predicate'] = predicate metric['value'] = value return 0 # if it is not there then just add it. self[0].append({'metricName': metric_name, 'predicate': predicate, 'value': value}) else: raise BadDictElement(metric_name, predicate, 'invalid predicate') def clear(self): """ avr_req['metricFilters'].clear() Clears metricFilters elements """ del self[0][:] class entityFilters(list): """ avr_req.['entityFilters'].add(dimension_name, predicate, values): All values are strings valid predicate is 'OPERATOR_TYPE_EQUAL' ['entityFilters'].clear() Clears the entityFilters element See rest_avr.ShowJsonApi for more details """ def __init__(self): self.append([]) def add(self, dimension_name, predicate, values): """ avr_req.['entityFilters'].add(dimension_name, predicate, values): All values are strings valid predicate is 'OPERATOR_TYPE_EQUAL' """ if predicate is 'OPERATOR_TYPE_EQUAL': # then loop throuth to see if the dimenson name already exists, if so replace for entity in self[0]: if entity['dimensionName'] == dimension_name: entity['predicate'] = predicate entity['values'] = values return 0 # if it is not there then just add it. self[0].append({'dimensionName': dimension_name, 'predicate': predicate, 'values': values}) else: raise BadDictElement(dimension_name, predicate, 'predicate must be OPERATOR_TYPE_EQUAL') def clear(self): """ ['entityFilters'].clear() Clears the entityFilters element """ del self[0][:] class reportFeatures(list): """ avr_req.['reportFeatures'].add( feature) adds report feature string. Multiple features are permitted. ['reportFeatures'].clear() Clears the analyticsModule element. See rest_avr.ShowAVRJsonApi for more details. """ def add(self, feature): """ avr_req.['reportFeatures'].add( feature) adds report feature string. Multiple features are permitted .""" if feature not in self: self.append(feature) def clear(self): """ ['reportFeatures'].clear() Clears the entityFilters element """ del self[:] class sortByMetrics(list): """ avr_req.['sortByMetrics'].add(metric_name, order) valid order names are 'ascending' and 'descending' sortByMetrics is optional in an AVR request. avr_req['sortByMetrics'].clear() Clears the sortByMetrics element. See rest_avr.ShowAVRJsonApi for more details. """ def __init__(self): self.metric_list = [] def add(self, metric_name, order): if metric_name not in self.metric_list: self.append({'metricName': metric_name, 'order': order}) self.metric_list.append(metric_name) def clear(self): """ ['sortByMetrics'].clear() Clears the sortByMetrics element """ del self[:] del self.metric_list[:] class viewDimensions(list): """ avr_req.['viewDimensions'].add(dimension_name): adds view dimension, only one dimension is allowed add will replace element if it already exists avr_req['viewDimensions'].clear() Clears the viewDimensions element. See rest_avr.ShowAVRJsonApi for more details. """ def __init__(self): self.append([]) self[0] = {} def add(self, dimension_name): """ avr_req.['viewDimensions'].add(dimension_name): adds view dimension string, only one dimension is allowed add will replace element if it already exists """ self[0]['dimensionName'] = dimension_name def clear(self, dimension_name): """ ['viewDimensions'].clear() Clears the viewDimensions element """ del self[0][:] class viewMetrics(list): """ avr_req.['viewMetrics'].add(metric_name): appends metric_name string to list. The specification allows multiple view metric elements avr_req['viewMetrics'].clear() Clears the viewMetrics elements See rest_avr.ShowAVRJsonApi for more details. """ def __init__(self): self.metric_list = [] def add(self, metric_name): """ avr_req.['viewMetrics'].add(metric_name): appends metric_name string to list. The specification allows multiple viewMetric elements """ if metric_name not in self.metric_list: self.append({'metricName': metric_name}) self.metric_list.append(metric_name) def clear(self): """ ['viewMetrics'].clear() Clears the viewMetrics elements """ del self[:] del self.metric_list[:] class timeRange(dict): """ avr_req.['timeRange'].add( t_from, t_to) both values are 16 digit numeric value in microseconds of unix/linux time. t_to is optional and can be replace by None timeRange is an optional. avr_req['timeRange'].clear() Clears the timeRange elements See rest_avr.ShowAVRJsonApi for more details. """ def add(self, t_from, t_to): """ avr_req.['timeRange'].add( t_from, t_to) both values are 16 digit numeric value in microseconds of unix/linux time. t_to is optional and can be replace by None timeRange is optional. """ if type(t_from) is long and len(str(t_from)) == 16: self['from'] = t_from else: raise BadTime(t_from + " is 16 digit numeric value in microseconds") if t_to != '' and t_to != 0 and t_to != None: if type(t_to) is long and len(str(t_from)) == 16: self['to'] = t_to else: raise BadTime(t_to + " is 16 digit numeric value in microseconds") else: if 'to' in self.keys(): del self['to'] def clear(self): """ ['timeRange'].clear() Clears the timeRange element """ del self[:] class pagination(dict): """ avr_req.['pagination'].add(num_results, skip_results) both are integer values. avr_req['pagination'].clear() Clears the pagination elements See rest_avr.ShowAVRJsonApi for more details. """ def add(self, num_results, skip_results): """ avr_req.['pagination'].add(num_results, skip_results) both arguments are integers. """ if type(num_results) is int: self['numberOfResults'] = num_results else: raise BadDictElement('number of Results ', num_results, 'must be integer') if type(skip_results) is int: self['skipResults'] = skip_results else: raise BadDictElement('skipResults ', skip_results, 'must be integer') def clear(self): """ ['pagination'].clear() Clears the pagination element """ del self[:] class avr_resp(dict): """ python response error is applicable. """ def __init__(self): self.error_layer = None self.error_code = None self.error_text = None class avr_req(dict): """ The main class for rest_avr. avr_req contains a dictionary that maps to the elements of a Icontrol REST AVR request along with capability of posting that request and receiving a response. The dictionary values are object instances of python classes that correspond to the the JSON values of the object's name/value pair. Each value has two public methods: avr_req.['objectName']add(): adds an element to the appropriate object with type checking. If an element allows more then one instance the add function will append the element If an element allows only one instance the add function will replace the element avr_req.['objectName'].clear()r: clears all elements in the object. printing rest_avr.ShowAVRJsonApi provides documentation for the AVR JASON elements. Further documentation is available on devcentral.f5.com To post an AVR Rest request there are two functions to populate the HTTP/HTTPS request. avr_req.auth(user, passw): provides the username and password avr_req.url_base(host, module) provides the host and the bigip module AVR queries to construct the URL to make the request. Then to post the request and return results in a python representation of the JSON response. avr_req.post_and_response() """ def __init__(self): self['analyticsModule'] = analyticsModule() self['analyticsModule'].parent = self self['pagination'] = pagination() self['metricFilters'] = metricFilters() self['entityFilters'] = entityFilters() self['reportFeatures'] = reportFeatures() self['sortByMetrics'] = sortByMetrics() self['viewDimensions'] = viewDimensions() self['viewMetrics'] = viewMetrics() self['timeRange'] = timeRange() self.avr_session = requests.session() self.avr_session.verify = False self.avr_session.headers.update({'Content-Type': 'application/json'}) # for multiple queued request handling. self.req_queue = [] self.generate_id = None self.done = None self.result = None self.num_requests = 0 self.res_queue = [] def post_and_response(self): """ returns a python representation of the json response to the request. failure returns array ['ERROR','component',error] """ warnings.filterwarnings("ignore") self.generate_request = self.avr_session.post(self.req_url_base + "/generate-report/", data=json.dumps(self)) self.generate_request_py = json.loads(self.generate_request.text) self.result_guid = self.generate_request_py['id'] self.results_status_url = self.req_url_base + "/generate-report/" + self.result_guid + "/?$select=status,reportResultsLink" self.results_url = self.req_url_base + "/report-results/" + self.result_guid self.sleeptime = .5 for i in range(5): time.sleep(self.sleeptime) self.sleeptime *= 2 # double backoff period each time. self.status_results_json = self.avr_session.get(self.results_status_url) self.status_results = json.loads(self.status_results_json.text) if self.status_results['status'] == 'FAILED': self.result = avr_resp() self.result_error_layer = 'REST' self.result_error_code = self.status_results['status'] self.result.error_text = self.status_results if self.status_results['status'] == 'FINISHED': self.raw_results_url = self.status_results['reportResultsLink'] self.results_url = self.raw_results_url.replace('localhost', self.host_name) self.results = self.avr_session.get(self.results_url) if self.results.status_code == 200: self.result = avr_resp() self.result.update(json.loads(self.results.text)) return self.result else: self.result = avr_resp() self.result.error_layer = 'HTTP' self.result.error_code = self.results.status_code self.result.error_text = self.results return self.result else: continue self.result = avr_resp() self.result.error_layer = 'REST_AVR' self.result.error_code = '408' self.result.error_text = 'TIMEOUT' def auth(self, user, passw): """ avr_req.auth(user, passw): username and password """ self.avr_session.auth = (user, passw) def url_base(self, host, module): """ avr_req.url_base(host, module) host and bigip module AVR queries to construct the URL to make the request. """ self.host_name = host self.req_url_base = 'https://%s/mgmt/tm/analytics/%s' % (host, module) self.module_py = {'analyticsModule': module} def add_to_queue(self): "adds request as currently constructed to queue" self.req_queue.append(deepcopy(self)) def clear_queue(self): """" clears request queue """ del self.req_queue[:] def post_and_response_queue(self): """ posts and sends response to from queue of requests. """ warnings.filterwarnings("ignore") for req in self.req_queue: req.generate_request = req.avr_session.post(req.req_url_base + "/generate-report/", data=json.dumps(req)) req.generate_request_py = json.loads(req.generate_request.text) req.generate_id = (req.generate_request_py['id']) req.results_status_url = self.req_url_base + "/generate-report/" + req.generate_id + "/?$select=status,reportResultsLink" self.sleeptime = .5 self.num_requests = len(self.req_queue) for i in range(5): for req in self.req_queue: if req.done is None: time.sleep(self.sleeptime) self.sleeptime *= 2 # double backoff period each time. req.status_results_json = req.avr_session.get(req.results_status_url) req.status_results = json.loads(req.status_results_json.text) if req.status_results['status'] == 'FAILED': req.result = avr_resp() req.result_error['layer'] = 'REST' req.result_error['error'] = req.status_results['status'] req.result_error['text'] = req.status_results if req.status_results['status'] == 'FINISHED': req.raw_results_url = req.status_results['reportResultsLink'] req.results_url = req.raw_results_url.replace('localhost', self.host_name) req.results = self.avr_session.get(req.results_url) if req.results.status_code == 200: req.result = avr_resp() req.result.update(json.loads(req.results.text)) req.done = True self.res_queue.append(req.result) self.num_requests -= 1 else: req.result = avr_resp() req.result_error.layer = 'HTTP' req.result_error.code = req.results.status_code req.result_error.text = req.results self.res_queue.append(req.result) if i == 5: if req.result == False: req.result = avr_resp() req.result.error_layer = 'REST_AVR' req.result.error_error = '408' req.result.error_text = 'TIMEOUT' if self.num_requests == 0: break return self.res_queue ShowAVRJsonApi = """ reportFeatures -------------- Specifies the kind of information that appears in a response from AVR. You may specify one or more of the following values: existing-entities time-aggregated time-series entities-count viewDimensions -------------- Specifies the dimensions for which to calculate a report, such as: {"dimensionName": "domain-name"} You may only specify a single dimension. You may omit this field in a report generation request. viewMetrics ----------- Specifies the list of metrics by which to sort results, such as: { "metricName": "average-tps" }, { "metricName": "transactions" } If you specify either time-aggregated or time-series features, you must specify one metric in a report generation request. sortByMetrics -------------- Specifies the list of metrics to sort by, such as: [{ metricName: "average-tps", order:"descending" } ] Valid values are ascending and descending. Sorting only applies to the time-aggregated feature. You do not need to specify this field in a report generation request. timeRange --------- Specifies the time range, in microseconds, for which to calculate a report, such as: {"from": 1410420888000000, "to": 1410424488000000 } You do not need to specify this field in a report generation request. entityFilters ============= Specifies the entities and values for which to calculate a report. You can specify a single entity with a second level of dimension filters that describe an aspect of the entity. If you specify multiple entity types, the results include only the entities that match all of the criteria. You do not need to specify this field in a report generation request. The following snippet contains two entities with corresponding values: [[{ "dimensionName" : "virtual", "predicate": "OPERATOR_TYPE_EQUAL", "values : ["phpAuction_VS_1"] }, { "dimensionName : "response-code", "predicate": "OPERATOR_TYPE_EQUAL", "values" : ["200"] } ]] metricFilters ------------- Specifies the metric filters for which to calculate a report, such as: [{ "metricName": "transactions", "predicate" : metricFilters "OPERATOR_TYPE_GREATER_THAN" "value": 100 }] You do not need to specify this field in a report generation request. For the existing-entities feature, AVR supports the OPERATOR_TYPE_LIKE predicate. AVR also supports the following predicates: OPERATOR_TYPE_EQUAL OPERATOR_TYPE_NOT_EQUAL OPERATOR_TYPE_GREATER_THAN OPERATOR_TYPE_LOWER_THAN OPERATOR_TYPE_GREATER_THAN_OR_EQUAL OPERATOR_TYPE_LOWER_THAN_OR_EQUAL pagination ---------- Specifies the number of results to return, and the number of results to skip, such as: { numberOfResults : 10, skipResults : 10} To see the second set of ten results, use the example shown here. AVR does not implement the OData query parameters top or skip. In order to see a specific set of results, you must set the number of results to return and then determine how many results to skip. You do not need to specify this field in a report generation request. """ Tested this on version: 12.0BIG-IP Report
Problem this snippet solves: Overview This is a script which will generate a report of the BIG-IP LTM configuration on all your load balancers making it easy to find information and get a comprehensive overview of virtual servers and pools connected to them. This information is used to relay information to NOC and developers to give them insight in where things are located and to be able to plan patching and deploys. I also use it myself as a quick way get information or gather data used as a foundation for RFC's, ie get a list of all external virtual servers without compression profiles. The script has been running on 13 pairs of load balancers, indexing over 1200 virtual servers for several years now and the report is widely used across the company and by many companies and governments across the world. It's easy to setup and use and only requires auditor (read-only) permissions on your devices. Demo/Preview Interactive demo http://loadbalancing.se/bigipreportdemo/ Screen shots The main report: The device overview: Certificate details: How to use this snippet: Installation instructions BigipReport REST This is the only branch we're updating since middle of 2020 and it supports 12.x and upwards (maybe even 11.6). Downloads: https://loadbalancing.se/downloads/bigipreport-v5.7.13.zip Documentation, installation instructions and troubleshooting:https://loadbalancing.se/bigipreport-rest/ Docker support https://loadbalancing.se/2021/01/05/running-bigipreport-on-docker/ Kubernetes support https://loadbalancing.se/2021/04/16/bigipreport-on-kubernetes/ BIG-IP Report (Legacy) Older version of the report that only runs on Windows and is depending on a Powershell plugin originally written by Joe Pruitt (F5) BIG-IP Report (only download this if you have v10 devices): https://loadbalancing.se/downloads/bigipreport-5.4.0-beta.zip iControl Snapin https://loadbalancing.se/downloads/f5-icontrol.zip Documentation and Installation Instructions https://loadbalancing.se/bigip-report/ Upgrade instructions Protect the report using APM and active directory Written by DevCentral member Shann_P: https://loadbalancing.se/2018/04/08/protecting-bigip-report-behind-an-apm-by-shannon-poole/ Got issues/problems/feedback? Still have issues? Drop a comment below. We usually reply quite fast. Any bugs found, issues detected or ideas contributed makes the report better for everyone, so it's always appreciated. --- Join us on Discord: https://discord.gg/7JJvPMYahA Code : BigIP Report Tested this on version: 12, 13, 14, 15, 16Logstash pipeline tester
Code is community submitted, community supported, and recognized as ‘Use At Your Own Risk’. Short Description A tool that makes developing logstash pipelines much much easier. Problem solved by this Code Snippet Oh. The problem... Have you ever tried to write a logstash pipeline? Did you suffer hair loss and splitting migraines? So did I. Presenting, logstash pipeline tester which givesyou a web interface where you can paste raw logs, send them to the included logstash instance and see the result directly in the interface. The included logstash instance is also configured to automatically reload once it detects a config change. How to use this Code Snippet TLDR; Don't do this, read the manual or checkout the video below Still here? Ok then! 🙂 Install docker Clone the repo Run these commands in the repo root folder:sudo docker-compose build # Skip sudo if running Windows sudo docker compose up # Skip sudo if running WindowsGo to http://localhost:8080on your PC/Mac Pick a pipeline and send data Edit the pipeline Send data Rince, repeat Version info v1.0.0 Docker containers no longer runs as root Vulnerability fix:https://github.com/epacke/logstash-pipeline-tester/releases/tag/v1.0.0 Video on how to get started: https://youtu.be/Q3IQeXWoqLQ Please note that I accidentally started the interface on port 3000 in the video while the shipped version uses port 8080. It took me roughly 5 hours and more retakes than I can count to make this video so that mistake will be preserved for the internet to laugh at. 🙂 The manual: https://loadbalancing.se/2020/03/11/logstash-testing-tool/ Code Snippet Meta Information Version: Check GitHub Coding Language: NodeJS, Typescript + React Full Code Snippet https://github.com/epacke/logstash-pipeline-testerConditional XOR operations
I was working on an iRule that only needed an OR if I didn’t reverse the logic on of them. But when I looked at the original problem, it needed one condition or the other but not both. You can nest conditionals to accomplish this, but I wanted an XOR operator, and Tcl doesn't have one at the string level, only with bitwise operations. So here is what that looks like in a truth table, where A could be an IP address range, B could be a hostname, and Q would be the truth condition. A B Q 0 0 0 0 1 1 1 0 1 1 1 0 In an iRule, this would look like this structurally (but you would need to add your comparisons for each A/B variable to whatever makes them true: if { ($a || $b) && !($a && $b) } { # act on the XOR true condition } else { # act on the XOR false condition }
GTM return LDNS IP to client
Problem this snippet solves: We do a lot of our load balancing based on topology rules, so it's often very useful to know where the DNS request is actually coming from rather than just the client's IP and the DNS servers they have configured. Especially if they're behind an ADSL router doing NAT or some other similar set up. This rule simply returns the IP address of the LDNS that eventually made the query to the GTM device in the response to a lookup for the WideIP using the rule, as well as logging the response and perceived location. Code : rule "DNS_debug" partition "Common" { when DNS_REQUEST { host [IP::client_addr] log local0.err "Debug address : [IP::client_addr] [whereis [IP::client_addr]]" } }
PowerShell module for the F5 LTM REST API
Problem this snippet solves: To report an issue with the F5-LTM or F5-BIGIP modules, please use the Issues sections of the GitHub repos (here and here) instead of commenting here. Thanks! This PowerShell module uses the iControlREST API to manipulate and query pools, pool members, virtual servers, and iRules. It aims to support version 11.5.1 and higher, and to conform to the schedule for technical support of versions, though this may eventually prove to become difficult. The module currently includes some functionality that, strictly speaking, is outside the scope of the LTM module. Hence, there is an active effort to wrap this LTM module into a larger BIG-IP module, and relocate that functionality elsewhere within that parent module, as well as expand the scope of functionality to include BIG-IP DNS (formerly GTM) and possibly other areas. Both the LTM module and the parent BIG-IP module are projects on github. Please use these projects to report any issues you discover. Thanks! The module contains the following functions. Add-iRuleToVirtualServer Add-iRuleToVirtualServer Add-PoolMember Add-PoolMonitor Disable-PoolMember Disable-VirtualServer Enable-PoolMember Enable-VirtualServer Get-CurrentConnectionCount (deprecated; use Get-PoolMemberStats | Select-Object -ExpandProperty 'serverside.curConns') Get-F5Session (will be deprecated in future versions. use New-F5Session) Get-F5Status Get-HealthMonitor Get-HealthMonitorType Get-iRule Get-iRuleCollection (deprecated; use Get-iRule) Get-Node Get-BIGIPPartition Get-Pool Get-PoolList (deprecated; use Get-Pool) Get-PoolMember Get-PoolMemberCollection (deprecated; use Get-PoolMember) Get-PoolMemberCollectionStatus Get-PoolMemberDescription (deprecated; use Get-PoolMember) Get-PoolMemberIP (deprecated; use Get-PoolMember) Get-PoolMembers (deprecated; use Get-PoolMember) Get-PoolMemberStats Get-PoolMemberStatus (deprecated; use Get-PoolMember) Get-PoolMonitor Get-PoolsForMember Get-StatusShape Get-VirtualServer Get-VirtualServeriRuleCollection (deprecated; use Get-VirtualServer | Where rules | Select -ExpandProperty rules) Get-VirtualServerList (deprecated; use Get-VirtualServer) Invoke-RestMethodOverride New-F5Session New-HealthMonitor New-Node New-Pool New-VirtualServer Remove-HealthMonitor Remove-iRule Remove-iRuleFromVirtualServer Remove-Pool Remove-PoolMember Remove-PoolMonitor Remove-ProfileRamCache Remove-Node Remove-VirtualServer Set-iRule Set-PoolLoadBalancingMode (deprecated; use Set-Pool) Set-PoolMemberDescription Set-Pool Set-VirtualServer Sync-DeviceToGroup Test-F5Session Test-Functionality Test-HealthMonitor Test-Node Test-Pool Test-VirtualServer How to use this snippet: To use the module, click 'Download Zip', extract the files, and place them in a folder named F5-LTM beneath your PowerShell modules folder. By default, this is %USERPROFILE%\Documents\WindowsPowerShell\Modules. The WindowsPowerShell and Modules folders may need to be created. You will most likely need to unblock the files after extracting them. Use the Unblock-File PS cmdlet to accomplish this. The Validation.cs class file (based on code posted by Brian Scholer) allows for using the REST API with LTM devices with self-signed SSL certificates. Nearly all of the functions require an F5 session object as a parameter, which contains the base URL for the F5 LTM and a credential object for a user with privileges to manipulate the F5 LTM via the REST API. Use the New-F5session function to create this object. This function expects the following parameters: The name or IP address of the F5 LTM device A credential object for a user with rights to use the REST API An optional TokenLifespan value for extending the life of the authentication token past the default 20 minutes You can create a credential object using Get-Credential and entering the username and password at the prompts, or programmatically like this: $secpasswd = ConvertTo-SecureString "PlainTextPassword" -AsPlainText -Force $mycreds = New-Object System.Management.Automation.PSCredential "username", $secpasswd Thanks to Kotesh Bandhamravuri and his blog entry for this snippet. There is a function called Test-Functionality that takes an F5Session object, a new pool name, a new virtual server, an IP address for the virtual server, and a computer name as a pool member, and validates nearly all the functions in the module. I've also contributed this code sample for how to gather some basic info about your LTM with this PS module. The module has been tested on: 11.5.1 Build 8.0.175 Hotfix 8 and later 11.6.0 Build 5.0.429 Hotfix 4 and later 12.0 / 12.1 13.0 Code : https://github.com/joel74/POSH-LTM-Rest Tested this on version: 11.5
iCR Python Module for iControl REST
Problem this snippet solves: This is a python module to simplify using iControl REST. Install using pip: pip install iCR or retrieve from https://pypi.python.org/pypi?:action=display&name=iCR&version=2.1 As simple as: #!/usr/bin/env python from iCR import iCR bigip = iCR("172.24.9.132","admin","admin") virtuals = bigip.get("ltm/virtual") for vs in virtuals['items']: print vs['name'] This prints out a list of Virtual Servers. Supported methods: init(hostname,username,password,[timeout,port,icontrol_version,folder,token,debug]) get(url,[select,top,skip,filter]) -> returns data or False getlarge(url,size,[select]) -> Used to retrieve large datasets in chunks. Returns data or False create(url,data) -> returns data or False modify(url,data,[patch=True]) -> returns data or False delete(url) -> returns True or False upload(file) -> file is a local file eg /var/tmp/test.txt, returns True or False download(file) -> files are located in /shared/images, returns True or False create_cert(files) -> files is an array containing paths to cert and key. Returns name of cert or False get_asm_id(name) -> name is the name of a policy. Returns an array of IDs or False create_hash(name) -> name is the name of the partition and policy. eg /Common/test_policy. This reduces the need to retrieve an array of hashes from the BIG-IP. Returns a string. get_token() -> this retrieves a BIG-IP token based on the username and password and sets it as the token in use. Returns the token ID or False delete_token() -> This deletes the object token from the BIG-IP and from the object create_transaction() -> creates a transaction and returns the transaction number ID as a string, or False. Subsequent requests will be added to thetransaction until commit_transaction is called. Transaction ID is stored in object.transaction commit_transaction() -> Commits the transaction stored in object.transaction. Returns True or False command(args,[cmd]) -> Runs a command using the arguments string args. Returns the returned output or True on success or False on failure. Note:Be sure to double-escape single quotes eg \\' and single escape double quotes eg \" cmd options are ping/save/load/restart/reboot Module Variables: icr_session - the link to the requests session raw - the raw returned JSON code - the returned HTTP Status Code eg 200 error - in the case of error, the exception error string headers - the response headers icontrol_version - set this to specify a specific version of iControl debug - boolean True or False to set debugging on or off port - set the port ( 443 by default ) folder - set this to create in a specific partition token - use this to set a specific token. If this is set, it will be used instead of basic auth select - use this with get to select the returned data top - use this with get to return a set number of records skip - use this to skip to a specific record number transaction - stores the Transaction ID How to use this snippet: Examples Setup a REST connection to a device #!/usr/bin/env python from iCR import iCR bigip = iCR("172.24.9.132","admin","admin",timeout=10) Create a Virtual Server vs_config = {'name':'test_vs'} createvs = bigip.create("ltm/virtual",vs_config,timeout=5) Retrieve the VS we just created virt = bigip.get("ltm/virtual/test_vs",select="name") print "Virtual Server created: " + virt['name'] Set the timeout bigip.timeout = 20 Now delete the VS we just created delvs = bigip.delete("ltm/virtual/test_vs") Retrieve ASM policy to ID mapping policies = bigip.get("asm/policies",select="name,id") Print a table of ASM policies with learning mode print print "Policy Name Learning Mode" print "------------------------------------------" for item in policies['items']: enabled = bigip.get("asm/policies/" + item['id'] + "/policy-builder",select="learningMode") print '{:32}'.format(item['name']) + enabled['learningMode'] File upload fp = "/home/pwhite/input.csv" if bigip.upload(fp): print "File " + fp + " uploaded" File download file="BIGIP-12.1.2.0.0.249.iso" download = bigip.download(file) if not download: print "File " + file + " download error" SSL Certificate creation In different folder bigip.folder = "TestFolder" files = ("TestCert.crt","TestCert.key") cert = bigip.create_cert(files) if cert: print "Certificate " + cert + " created" Turn on debugging bigip.debug = True Retrieve ASM policy IDs asm = bigip.get_asm_id("dummy_policy") print len(asm) + " IDs returned" print "ID: " + str(asm[0]) Convert an ASM policy name to hash hash = bigip.create_hash("/Common/test-policy") enabled = bigip.get("asm/policies/" + hash + "/policy-builder",select="learningMode") print '{:32}'.format(item['name']) + enabled['learningMode'] Retrieve and use a token bigip.get_token() Delete the token bigip.delete_token() Developed on Python 2.7 but works with v3. Works on TMOS 11.6 onwards though some features may not be implemented, such as tokens. If you use this and have found bugs, would like to discuss it or suggest features then please PM me on DevCentral. Tested this on version: 13.0F5 XC reviewing API requests which the GUI sends and a backup of the config with Python/Ansible
Short Description The F5 XC Distributed Cloud GUI in the background sends API requests with JSON body to the system and those requests can be easily reviewed. Problem solved by this Code Snippet If someone wonders how to do some tasks that the XC GUI does the same way but with automation through the API and JSON then this article will help them. Also at the end I have shown how to retrive XC json data with API. How to use this Code Snippet Reviewing the API requests that are generated by the XC GUI. Full Code Snippet There are 3 ways to review the API requests that the GUI generates. On each XC element like for example the load balancer you can click on the JSON and see the JSON code. The JSON code can even be directly edited from the GUI Dashboard! The API documentation can be reviewed directly from the XC GUI. The final option is just to use the browser developer tools and to see what API requests are send by the F5 XC. This feature is now present on most F5 new products like F5OS(Velos/rSeries) and F5 NEXT😉 The XC JSON created objects from the API are a form of a backup configuration. Even if the objects were created from the GUI then API GET requests can be used to retrive their JSON data and this can be saved to a backup file in the form of snapshot. I have used Python with requests library and the url and API key are added as user input arguments. The script can be used to get information like the XC LB or service policies. As example "/api/config/namespaces/default/service_policys". The script will first call an API endpoint to get for example all the load balancer or service policy names and then it will use the names get the config for each individual service policy or load balancer, using a for loop. There is time.sleep(1) to add 1 second slowness between each api request. The code can have the full url like https://{tenant_name}.console.ves.volterra.io/api/config/namespaces/{namespace}service_policysand the api token be added during script execution or the arguments can be input at the start of script execution by commenting outurl = sys.argv[1] and api_token = sys.argv[2] and executing the script like python3 service_policy.py {argument1} {argument2}. The api token by default is hidden using the getpass library for extra security. See Github for the code: Nikoolayy1/xc_api_script: XC API script to retrieve basic json config (github.com) In some cases using Terraform for XC will be best as XC has strong Terraform support as seen at the link below. Ansible can also be used but XC does not have many developed Ansible modules, so in many cases the Ansible URI module will need to be used and the Ansible URI module (ansible.builtin.uri module – Interacts with webservices — Ansible Documentation) in the backgroung is just the python requests or http.client module, as Ansible is python in the background, so better use python directly in that case. XC Terraform: Terraform | F5 Distributed Cloud Tech Docs F5 Distributed Cloud WAAP deployment with Terrafor... - DevCentral Using Terraform and F5® Distributed Cloud Mesh to ... - DevCentral Example Ansible code (even if I said python is better in this case 😀) xc_api_script/xc_ansible at main · Nikoolayy1/xc_api_script (github.com)Session Table Export
Problem this snippet solves: This sample goes along with the Tech Tip titled Session Table Exporting With iRules . It creates a mechanism for you to export the data from your session tables for archiving or external reporting. NOTE: This functionality is included in the Session Table Control iRule and is partially rendering here so it has been removed.
