Parsing tmsh command output with Python TextFSM module and some Lessons learned
It’s been a while! I’m back with some code to parse tmsh command output, extracting specific sections of the output and present it in easy-to-read format.
In my previous article, BIG-IP Advanced Firewall Manager (AFM) DNS NXDOMAIN Query Attack Type Walkthrough, I intended to gather DNS statistics and packet captures from the BIG-IP, use the observed stats to understand the DNS traffic, and use it as basis for detection and mitigation threshold for BIG-IP AFM DNS NXDOMAIN Query Attack Types.
With the same goal, my use case is parsing DNS statistics from the command output tmsh show ltm profile dns <DNS profile>. I'm interested in the "A" Question Type and "NXDOMAIN" Response Message Return Code (RCODE) values.
TextFSM
TextFSM is a library used to parse semi-structured text data into structured data using templates that define patterns to match.
As I'm using VSCode, here is the description of the textfsm module:
=============
(module) textfsm
Template based text parser.
This module implements a parser, intended to be used for converting human readable text, such as command output from a router CLI, into a list of records, containing values extracted from the input text.
A simple template language is used to describe a state machine to parse a specific type of text input, returning a record of values for each input entity.
=============
I will be using TextFSM python module to create a template where regular expressions are defined to match values of interest and store them into records.
I created 4 files and can be found in https://github.com/arvfopa/scripts
dns_stat.txt is the output of the bash script that gather periodically the tmsh show ltm profile dns command output. Here's the sample:
===========
while true; do date >> dns_stat.txt; tmsh show ltm profile dns {dns profile name} >> dns_stat.txt; sleep 20; echo "#######" >> dns_stat.txt; done
===========
In this dns stat file, I wanted to extract the Date the command was run, the value of the A (host) record in the Question Type section and the No Name (NXDOMAIN) value in the Return Code (RCODE) section.
https://github.com/arvfopa/scripts/blob/main/dns_stat.txt
dns_t_1column.txt is the TextFSM template where I defined regular expressions to match the values of interest.
https://github.com/arvfopa/scripts/blob/main/dns_t_1column.txt
=======
Value DATE (\S+.*)
Value DNS_Q_typeA (\d+)
Value DNS_R_NXDOMAIN (\d+)
Start
^Date: ${DATE}
^\s+A\s+${DNS_Q_typeA}
^Return Code \(RCODE\)
^\s{3}No Name \(NXDOMAIN\)\s+${DNS_R_NXDOMAIN} -> Record
=======
"Value" Definitions
These lines define variables (values) that will capture specific parts of the text based on the provided regular expressions (regex).
"Start" State
This section defines the start state of the FSM (Finite State Machine) and the patterns it should match to transition states and capture values.
As in my previous article, I also used ChatGPT a lot in creating and testing the scripts. When I had questions, I simply pasted the code and ChatGPT gives the explanation such as when I was trying to figure out the regular expression to match specifically the values in the dns stat sample file - which text and values, how many spaces in between. ChatGPT was very handy in the coding process especially when I can just prompt it to "explain".
textfsm_dns_stat_parse.py is the python script which imports the textfsm module and use the TextFSM template created to parse the dns stat sample file. The parsed values are then tabulated.
https://github.com/arvfopa/scripts/blob/main/textfsm_dns_stat_parse.py
===========
import textfsm
from tabulate import tabulate
with open('dns_t_1column.txt') as template, open('dns_stat.txt') as dnsstat:
re_table = textfsm.TextFSM(template)
header = re_table.header
result = re_table.ParseText(dnsstat.read())
tabulated = (tabulate(result, headers=header))
print(tabulated)
with open('fsmtabulated.txt', mode='w', newline='') as file:
file.write(tabulated)
===========
fsmtabulated.txt is the output file of the textfsm_dns_stat_parse.py where the extracted values from the dns stat sample file are neatly tabulated.
https://github.com/arvfopa/scripts/blob/main/fsmtabulated.txt
As observed from the tabulated output, DNS_Q_typeA values refer to the A type DNS queries and DNS_R_NXDOMAIN are the DNS NXDOMAIN responses. Both values show an increase almost proportionally.
For a BIG-IP administrator, an increase of DNS A queries may be expected, however, if the DNS NXDOMAIN response also increases, it may be that the DNS listener Virtual Server that uses the DNS profile is receiving a flood of DNS A queries for hostnames that do not exist in the backend DNS server pool and consumes resources - CPU, memory and bandwidth - and overwhelm the DNS servers.
BIG-IP AFM have DNS DoS Attack Vectors that can be configured to detect and mitigate DNS DoS attacks. Making use of the dns stat data, we can configure detection and mitigation thresholds for DNS A Query and DNS NXDOMAIN Query to detect at 4000 EPS (events per second) and mitigate at 6000 EPS. Exceeding packets on the detected and mitigated vector will be dropped until the mitigation threshold is not exceeded.
This will prevent excess DNS queries of certain types from reaching the backend DNS server and keep the service available.
Lessons Learned
In the previous Parsing F5 BIG-IP LTM DNS profile statistics and extracting values with Python article, I created a python script that sent iControl REST request thru the BIGREST module to a BIG-IP for DNS stats from a VS with a DNS profile.
Regarding this error:
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='IP address', port=443): Max retries exceeded with url: /mgmt/shared/echo-query (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1007)')))
The "certificate verify failed: self signed certificate" error can be side stepped by setting the "session_verify=False" of the BIGIP class of the BIGREST module. While this is possible, using a TLS certificate on the BIG-IP that is trusted by the client that will run the python script is the secure option.
Procedure on creating TLS device certificate for the BIG-IP can be found in these articles.
K9114: Creating a new SSL device certificate and key pair
K6353: Updating a self-signed SSL device certificate on a BIG-IP system
Import the BIG-IP device certificate to the client's trusted certificate store or if the BIG-IP device TLS certificate is signed by a trusted CA, the client should be able to verify the BIG-IP device TLS certificate.
On the python script, the variable that requests the stats from the DNS profile used in the DNS listener VS value is set to the URI for the specific VS name and the DNS profile.
data = device.show("/mgmt/tm/ltm/virtual/dns-10.73.125.137/profiles/~Common~dns-p-1/")
The script can be modified to reference a different VS and a different DNS profile configured on the VS
data = device.show("/mgmt/tm/ltm/virtual/<DNS Listener VS name>/profiles/<DNS Profile>/")
Conclusion
Gathering periodic data from tmsh output and parsing it using tools such as python scripts can help BIG-IP administrators as it presents the collected stats in a human friendly format, instead of scrolling thru many lines of tmsh command output. The observed values can be then used to make configuration decisions as in the use case for this python script such as configuring detection and mitigation thresholds for DNS A Query and DNS NXDOMAIN Query to detect and mitigate DNS DoS attacks. Lastly, while there are existing tools/reporting - example, in the BIG-IP GUI, one can go to Statistics ›› Module Statistics : DNS : Delivery ›› GTM Listeners : <DNS VS Listener> and observe the profile statistics including from DNS profiles. This exercise offers alternative data gathering and analysis option for a BIG-IP administrator.
I hope this article has been educational.
The F5 SIRT creates security-related content posted here in DevCentral, sharing the team’s security mindset and knowledge. Feel free to view the articles that are tagged with the following: