Parsing F5 BIG-IP LTM DNS profile statistics and extracting values with Python

Introduction

Hello there! Arvin here from the F5 SIRT. 

A little while ago, I published F5 BIG-IP Advanced Firewall Manager (AFM) DNS NXDOMAIN Query Attack Type Walkthrough part one and two, where I went through the process of reviewing BIG-IP LTM DNS profile statistics and used it to set BIG-IP AFM DNS NXDOMAIN Query attack type detection and mitigation thresholds with the goal of mitigating DNS NXDOMAIN Floods. 

In this article, I continue to look at BIG-IP LTM DNS profile statistics, find ways of parsing it and to extract specific values of interest through Python. 

Python for Network Engineers

Python has emerged as a go-to language for network engineers, providing a powerful and accessible toolset for managing and automating network tasks. Known for its simplicity and readability, Python enables network engineers to script routine operations, automate repetitive tasks, and interact with network devices through APIs. With extensive libraries and frameworks tailored to networking, Python empowers engineers to streamline configurations, troubleshoot issues, and enhance network efficiency. Its versatility makes it an invaluable asset for network automation, allowing engineers to adapt to evolving network requirements and efficiently manage complex infrastructures. Whether you're retrieving data, configuring devices, or optimizing network performance, Python simplifies the process for network engineers, making it an essential skill in the modern networking landscape.  

The Tools

ChatGPT3.5

The "Python for Network Engineers" intro came from ChatGPT3.5 [:)].  Throughout this article, the python coding "bumps" avoidance and approaches came from ChatGPT3.5. Instead of googling, I asked ChatGPT "a lot" so I could get the python scripts to get the output I wanted.

https://chat.openai.com/

Visual Studio Code

Using Visual Studio Code (VSCode) to build the scripts was very helpful, especially the tooltip / hints which tells me and help make sense of the available options for the modules used and describing the python data structures.

Python 3.10

(From ChatGPT) Python 3.10, the latest version of the Python programming language, brings forth new features and optimizations that enhance the language's power and simplicity. With Python's commitment to readability and ease of use, version 3.10 introduces structural pattern matching, allowing developers to express complex logic more concisely. Other improvements include precise types, performance enhancements, and updates to syntax for cleaner code. Python 3.10 continues to be a versatile and accessible language, serving diverse needs from web development to data science and automation. Its vibrant community and extensive ecosystem of libraries make Python 3.10 a top choice for developers seeking both efficiency and clarity in their code.

 

Here is the sample script output

Here is the sample CSV-formatted data in a csv file with timestamp of the script run

I created a github repository for the Python script and its sample script output and csv data

see https://github.com/arvfopa/scripts/tree/main

https://github.com/arvfopa/scripts/blob/main/extractAreq - Python Script "extractAreq"

https://github.com/arvfopa/scripts/blob/main/extractAreq_output - "extractAreq" output

Bumps along the way

BIGREST module

I initially encountered an error 'certificate verify failed: self signed certificate' when provided only the IP address and credentials used in the BIGIP class of the bigrest.bigip python module

raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='IP address', port=443): Max retries exceeded with url: /mgmt/shared/echo-query (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1007)')))

This is fixed by setting "session_verify" argument of the BIGIP class to "false"  to  disables SSL certificate validation

https://bigrest.readthedocs.io/utils.html

I also received this error "TypeError: Object of type RESTObject is not JSON serializable"

raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type RESTObject is not JSON serializable

I reread the BIGREST documentation and found that the output is a python dictionary and can is printed in json format. I rechecked the script and removed the json related syntax and module, and the script runs fine and still gets the same output. I updated the script on github with the simplified changes.

https://bigrest.readthedocs.io/restobject.html

Here's a sample of the RESTObject properties dictionary values. Plenty of data can be extracted. Example, "clientside.pktsIn" value, a virtual server statistic, can be observed and should detection and mitigation thresholds for AFM, say,  UDP protocol DoS attack type, need to be set. This value can be monitored over time to understand how many packets a virtual server receives. 

==============

{'clientside.bitsIn': {'value': 0}, 'clientside.bitsOut': {'value': 0}, 'clientside.curConns': {'value': 0}, 'clientside.evictedConns': {'value': 0}, 'clientside.maxConns': {'value': 0}, 'clientside.pktsIn': {'value': 0}, 'clientside.pktsOut': {'value': 0}, 'clientside.slowKilled': {'value': 0}, 'clientside.totConns': {'value': 0}, 'cmpEnableMode': {'description': 'all-cpus'}, 'cmpEnabled': {'description': 'enabled'}, 'csMaxConnDur': {'value': 0}, 'csMeanConnDur': {'value': 0}, 'csMinConnDur': {'value': 0}, 'destination': {'description': '10.73.125.137:53'}, 'ephemeral.bitsIn': {'value': 0}, 'ephemeral.bitsOut': {'value': 0}, 'ephemeral.curConns': {'value': 0}, 'ephemeral.evictedConns': {'value': 0},

==============

CSV filename issue

I encountered this error, "OSError: [Errno 22] Invalid argument: 'dns_stats_2023-12-07_18:01:11.csv". This is related to writing of the output csv file.  I asked ChatGPT what this was about and was provided with this answer.

=======================

The error you're encountering, "[Errno 22] Invalid argument," typically suggests an issue with the filename or file path. In this case, it seems to be related to the colon (':') character in the filename.

In some operating systems (like Windows), certain characters are not allowed in filenames, and ":" is one of them. Since you're including a timestamp in the filename, it's common to replace such characters with alternatives.

You can modify the timestamp format to use underscores or hyphens instead of colons.

====================

The timestamp variable in the script stores the value of the formatted timestamp that will be used in the filename. It initially used a colon (:) as the hour/min/sec separator. It was changed to dash (-) so it would not encounter this error.

timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")

Checking the file csv file if it exists

The function "write_to_csv" writes the time of collection (formatted_date) and extracted value of DNS A requests count (AReqsvalue). It is called every 10 seconds [time.sleep(10)] for a minute [end_time = time.time() + 60] and writes the output to a file in csv format. The "tabulate" function formats the output of the script. Getting the arrangement of the execution wrong would result in unexpected output. The "file_exists" check to write the "headers" was added to make sure that the "headers" are only written once. 

"write_to_csv" function

========================

def write_to_csv(formatted_date, AReqsvalue):
current_datetime = datetime.now()
formatted_date = current_datetime.strftime("%Y-%m-%d %H:%M:%S")

csv_filename = f"dns_stats_{timestamp}.csv"
  headers = ["Date", "DNS A requests"]
  stats = [[formatted_date, AReqsvalue]]

  file_exists = os.path.exists(csv_filename)

  print(tabulate(stats, headers, tablefmt="fancy_grid"))

  with open(csv_filename, mode='a', newline='') as file:
    writer = csv.writer(file)

    if not file_exists:
      writer.writerow(headers)

writer.writerows(stats)

end_time = time.time() + 60

while time.time() < end_time:
write_to_csv(formatted_date, AReqsvalue)
time.sleep(10)

==========================

 Using ChatGPT

In building this script, I used ChatGPT "a lot" and it helped to provide make more sense of the module options, errors and sample scripts. It has been a helpful tool. 

It tracks your conversation/questions to it and kind of understands the context/topic.

"ChatGPT can make mistakes. Consider checking important information." is written at the bottom of the page.

The data I used in this article are data from a lab environment. That said, when using public AI/ML systems, we should ensure we do not send any sensitive, proprietary information. Organizations have rolled out their own privacy policies when using AI/ML systems, be sure to follow your own organization's policies.

Conclusion

Using python to parse and extract values of interest from LTM profile statistics offers flexibility and hopefully simplifying observing and recording these data for further use. In particular, setting values for BIG-IP AFM DoS Detection and Mitigation thresholds will be easier if such data has been observed as it, in my opinion, is the "pulse" of the traffic the BIG-IP processes. As noted in the sample json data output, we can see many statistics that can be reviewed and observed to make configuration changes relevant, for example, mitigating a connection spike by setting a VS connection/rate limit. We can look at the "Conns" values and use the observed values to set a connection limit. 

Example:

'clientside.curConns': {'value': 0}, 'clientside.evictedConns': {'value': 0}, 'clientside.maxConns': {'value': 0},  'clientside.totConns':{'value': 0}

That's it for now.  I hope this article has been educational.

The F5 SIRT creates security-related content posted here in DevCentral, sharing the team's security mindset and knowledge. Feel free to view the articles that are tagged with the following:

 

• F5 SIRT
• series-F5SIRT-this-week-in-security
• TWIS