HTTP Brute Force Mitigation Playbook: Bad Actor Behavior and Gathering Statistics using BIG-IP LTM Policies, iRules and BIG-IP AVR - Chapter 2
Gathering Statistics and Bad Actor Behavior In this Chapter of the HTTP Brute Force Mitigation Playbook series, we will review BIG-IP Local Traffic Manager (LTM) and Application Visibility and Reporting (AVR) modules' features to show how it can used to gather statistics and how we can use these statistics as a base line of what an organization may consider as normal traffic and malicious traffic and how it relates to HTTP Brute Force Attacks. Normal Traffic and Malicious Traffic Definition Normally, we consider normal traffic as client activities associated to an application that is expected and does not affect availability and performance. Malicious traffic, on the other hand, are traffic bound to hit any public facing application or device with the goal of impacting availability, performance and its security. We will focus on malicious traffic related to HTTP Brute Force Attacks. Bad Actor Behavior In the case of HTTP Brute Force Attacks, it is common to observe these considered Bad Actor Behavior from malicious traffic: A spoofed HTTP User-Agent header Traffic from unexpected Geolocation(s) Excessive traffic from a certain source IP Multiple Login attempts on a web application from the same or distributed source IP using different credentials HTTP requests to an application that iterates thru a HTTP parameter's value either from a single or distributed traffic source The malicious traffic could be sent slowly and in low volume in an attempt to be more stealthy and applying on or more of the Bad Actor Behavior combinations or also be sent thru high volume traffic and will have noticeable impact such as high CPU usage on the application server Prerequisites Managing the BIGIP configuration requires Administrator access. Ensure access to Configuration Utility (Web GUI) and SSH is available. These management interfaces will be helpful in configuring, verifying and troubleshooting on the BIGIP.Having access to serial console output of the BIGIP is also helpful. When working with F5 Support, having an F5 Support/AskF5 account will be helpful.To access your AskF5 account, access the following link: https://support.f5.com/csp/my-support/home Another helpful site is F5 ihealth, where qkviews extracted from the BIGIP device can be uploaded and analysed. Logs, statistics, heuristics, graphs and BIGIP configuration extracted thru a qkview are available in ihealth when uploaded and will aid in diagnosing and troubleshooting efforts. https://ihealth.f5.com/ Register an account to these F5 Sites as these will helpful in working with F5 Support.When troubleshooting, generating a qkview or extracting bigip logs or viewing configuration in the GUI require access to the BIGIP and having the administrative access to it and referenced F5 Support sites will be helpful. Local Traffic Manager (LTM) and Application Visibility and Reporting (AVR) license are required to use the related features. LTM and AVR modules should be provisioned to access related configuration discussed in this chapter. Using Local Traffic Manager (LTM) features to gather HTTP request data Local Traffic Manager (LTM)features such as iRules and LTM Policy can be used to log messages generated from a certain stage of the HTTP request. In the case of a HTTP brute force attack, inspecting the source IP, HTTP headers, the HTTP request payload can provide information on what malicious traffic may look like. Required License and Provisioning BIG-IP LTM module should licensed and provisioned for LTM features to be available. A "harmless" HTTP request The following "curl" command will receive a proper response from a web site it is requesting content from. curl --silent --output /dev/null -lvk https://172.16.8.84 -H "User-Agent: im-a-web-browser" The web server does not validate whether this curl request is coming from a legitimate web browser as a normal user would use.the curl "-H" option allows a HTTP header to be included in the curl HTTP request, in this case "User-Agent: im-a-web-browser" will be the HTTP User-Agentand value that the web server will receive. A legitimate HTTP User-Agent value can also be used for this option and the web server will simply accept and process the request without validating whether the request is from a real web browser or an automated client . Using Local Traffic Manager (LTM) Policy to log HTTP request data A Local Traffic Manager (LTM) Policy can be applied to a Virtual Server and match a traffic pattern and execute defined actions. LTM Policy configuration is straightforward and is able to log essential data, such as in the case of a HTTP Brute Force attack, gather source IP, HTTP headers and HTTP payload. It can log locally or remotely. Here is a sample LTM Policy configuration where it will log the Source client IP address and port, HTTP header names and its interesting values and the HTTP Payload for all traffic at HTTP request time. From the BIG-IP Configuration Utility: Local Traffic››Policies : Policy List››/Common/log_http_traffic:log_http_request_header_payload log_http_traffic is the name of the LTM Policy log_http_request_header_payload is a rule in the log_http_traffic LTM Policy This view is only in Draft state of the LTM Policy These are configured in the message section of the "Log" action lines. notice the second Log action specifically logs the User-Agent and X-Forwarded-For HTTP headers. Common in HTTP Brute Force Attacks, the HTTP User-Agent header's value may be a slightly modified compared to a legitimate User-Agent value - imitating a legitimate web browser. Similarly, the X-Forwarded-For value may provide some hints where the attack traffic may really be coming from. In scenarios where the attacker may be passing thru web proxies, reviewing the X-Forwarded-For header value can be helpful in determining possible thru source of the attack traffic. Log action 1: tcl: client [IP::client_addr]:[TCP::client_port] -> URL: [HTTP::host][HTTP::uri]: http headers are [HTTP::header names] and the http payload is [HTTP::payload] Log action 2: tcl: client [IP::client_addr]:[TCP::client_port] -> URL: [HTTP::host][HTTP::uri]: interesting HTTP headers are User-Agent:[HTTP::header User-Agent] X-Forwarded-For [HTTP::header X-Forwarded-For] Notice that the Log action references "tcl". LTM Policy actions accepts tcl expression and is commonly used in iRules. iRule commands such as the[HTTP::header] can be used in LTM policies. Important: The iRule commands used in this sample LTM policy are for demonstration purpose only. Complex LTM Policies with iRule commands should be properly tested. F5 Professional Services is the best resource for this type of implementation. log Facility local0 refers to the /var/log/ltm and thus, generated logs will be in /var/log/ltm for this LTM Policy. other log files can be also specified and remote logging is also possible. Here is the Published LTM Policy The sample LTM Policy is then applied to a Virtual Server where malicious traffic is suspected to be hitting Here is the text form of the Published LTM Policy. log_http_request_header_payload 1. Log message 'tcl: client [IP::client_addr]:[TCP::client_port] -> URL: [HTTP::host][HTTP::uri]: http headers are [HTTP::header names] and the http payload is [HTTP::payload]' at request time. 2. Log message 'tcl:client [IP::client_addr]:[TCP::client_port] -> URL: [HTTP::host][HTTP::uri]: interesting HTTP headers are User-Agent:[HTTP::header User-Agent] X-Forwarded-For:[HTTP::header X-Forwarded-For]' at request time. log generated in /var/log/ltm Nov 29 15:24:22 BIG-IP info tmm7[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.7.31:39063 -> URL: 172.16.8.86/: http headers are Host Accept User-Agent X-Forwarded-For Content-Length Content-Type and the http payload is admin:admin Nov 29 15:24:22 BIG-IP info tmm7[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.7.31:39063 -> URL: 172.16.8.86/: interesting HTTP headers are User-Agent:im-a-web-browser X-Forwarded-For:88.88.88.88 Using grep, awk and uniq in the BIG-IP bash prompt, /var/log/ltm can be inspected to check the how frequent a source client IP sent a HTTP request and along with it, HTTP header and Payload information, to the Virtual Server where the LTM Policy is applied. Here, the sample grep/awk/uniq command will count the number of times the client source IP was observed from the sample LTM Policy generated log /var/log/ltm. In the sample LTM Policy, it logs the client source IP twice, as such the sample grep/awk/uniq command was written to take account of this. grep/awk/uniq sample command 1: grep client /var/log/ltm | awk -F " " '{print $9}' | uniq -c | awk -F ":" '{print $1}' | awk -F " " '{print $2}' | uniq -c sample output: root@BIG-IP:Active:Standalone] config # grep client /var/log/ltm | awk -F " " '{print $9}' | uniq -c | awk -F ":" '{print $1}' | awk -F " " '{print $2}' | uniq -c 9 172.16.7.31 grep/awk/uniq sample command 2: grep interesting /var/log/ltm | awk -F " " '{print $18}' | awk -F ":" '{print $2}' | uniq -c This sample command will count the number of time the X-Forwarded-For HTTP Header appeared in the log. the word 'interesting' was initially filtered as it is the line where theX-Forwarded-For information was logged. sample output: root@BIG-IP:Active:Standalone] config # grep interesting /var/log/ltm | awk -F " " '{print $18}' | awk -F ":" '{print $2}' | uniq -c 20 88.88.88.88 The sample grep/awk/uniq command are just simple demonstration commands only. You can build your own scripts, version of the command to get the desired information.The sample output are also for demonstration purpose only. In a real HTTP brute force attack, The sample LTM Policy to log the client source IP and HTTP information in the request will be helpful in describing the possibly malicious traffic. Using F5 iRules to gather HTTP request data From the F5 iRule Home Page, https://clouddocs.f5.com/api/irules/: F5 iRules is a powerful and flexible feature within the BIG-IP® local traffic management (LTM) system that you can use to manage your network traffic. The iRulesTM feature not only allows you to select pools based on header data, but also allows you to direct traffic by searching on any type of content data that you define. Thus, the iRules feature significantly enhances your ability to customize your content switching to suit your exact needs. iRules can also be used to log source IP and HTTP information. In comparison to LTM Policy, iRules are much more flexible and can also be very complex. Depending on how much information you would like to gather during a HTTP Brute Force attack, iRules can be very verbose. iRules are also resource intensive, the more it gets executed as traffic hits a Virtual Server where it is applied, it will consume additional CPU resource as well. If the same information can be gathered using LTM Policy feature, use it instead of iRules. For simplicity we will gather the same HTTP request data as demonstrated in the LTM Policy section. Important: The iRule commands used in the sample iRules are for demonstration purpose only. Complex iRules should be properly tested. F5 Professional Services is the best resource for this type of implementation. In the Configuration Utility,Local Traffic››iRules : iRule List›› can be created. Sample iRule, log_http_request_header_payload The iRule is then applied to a Virtual Server where suspected malicious traffic is hitting Sample iRule: # Log debug to /var/log/ltm? 1=yes, 0=no when RULE_INIT { set static::payload_dbg 1 } when HTTP_REQUEST { set LogString "Client [IP::client_addr]:[TCP::client_port] -> [HTTP::host][HTTP::uri]" if {$static::payload_dbg}{log local0.debug "=============================================" } if {$static::payload_dbg}{log local0.debug "$LogString (request)"} # log each Header. foreach aHeader [HTTP::header names] { if {$static::payload_dbg}{log local0.debug "$aHeader: [HTTP::header value $aHeader]"} } if {$static::payload_dbg}{log local0.debug "============================================="} if {[HTTP::header "Content-Length"] ne "" && [HTTP::header "Content-Length"] <= 1048000} { HTTP::collect [HTTP::header "Content-Length"] } else { HTTP::collect 1048000 } } when HTTP_REQUEST_DATA { # Log the bytes collected if {$static::payload_dbg}{log local0.debug "Collected [HTTP::payload length] bytes"} # Log the payload locally if {[HTTP::payload length] < $static::max_chars}{ if {$static::payload_dbg}{log local0.debug "Payload=[HTTP::payload]"} } } Sample log output in /var/log/ltm: Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: ============================================= Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: Client 172.16.7.31:10761 -> 172.16.8.86/ (request) Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: Host: 172.16.8.86 Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: Accept: */* Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: User-Agent: im-a-web-browser Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: X-Forwarded-For: 88.88.88.88 Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: Content-Length: 11 Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: Content-Type: application/x-www-form-urlencoded Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: ============================================= Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST_DATA>: Collected 11 bytes Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST_DATA>: Payload=admin:admin Using the same grep/awk/uniq commands, we can observe the number of times a client has sent a HTTP request to a Virtual Server using the sample iRule. grep/awk/uniq sample command 3: grep "<HTTP_REQUEST>: Client" /var/log/ltm | awk -F " " '{print $11}' | awk -F ":" '{print $1}' | uniq -c sample output: root@BIG-IP:Active:Standalone] config # grep "<HTTP_REQUEST>: Client" /var/log/ltm | awk -F " " '{print $11}' | awk -F ":" '{print $1}' | uniq -c 13 172.16.7.31 grep/awk/uniq sample command 4: grep "<HTTP_REQUEST>: User-Agent" /var/log/ltm | awk -F " " '{print $11}' | uniq -c The sample command will count the number of HTTP requests a specific User-Agent has sent to a Virtual Server sample output: root@BIG-IP:Active:Standalone] config # grep "<HTTP_REQUEST>: User-Agent" /var/log/ltm | awk -F " " '{print $11}' | uniq -c 13 im-a-web-browser Additional Notes for LTM Policy and iRules HTTP request data gatheringfor HTTP Brute Force attack The sample iRule and LTM Policy provided are just sample approach in observing the traffic to a Virtual Server suspected of being attacked during a HTTP Brute Force attack.There are more LTM Policy and iRule events that can be used to gather data and observe it to qualify certain characteristics of the traffic that can be considered malicious. In both the LTM Policy and iRule samples, we logged HTTP Payload output. During a HTTP Brute Force attack, an attacker - perhaps characterized by a certain source IP or a suspicious User-Agent value,may be sending to the HTTP based application random username and password or, legitimate looking credentials which are normally extracted from a breach,in attempt to guess a legitimate user credentials to compromise and to further do damage. The generated logs can then be used for your analysis and decision making steps in the mitigation of the attack. While the logs that will be generated are useful during a HTTP Brute Force attack, these logs may be excessive and may not be needed when there is noHTTP Brute Force attack event. Removing the sample iRule or LTM Policy from the Virtual Server would prevent further logging. Using F5 Application Visibility and Reporting to gather HTTP request data From the BIG-IP Analytics: Implementations Manual: https://techdocs.f5.com/kb/en-us/products/big-ip_analytics/manuals/product/big-ip-analytics-implementations-14-1-0/01.html#guid-ce2cb6f5-24b4-4570-a827-b8b4db8e19e8 Analytics, or Application Visibility and Reporting (AVR), is a module on the BIG-IP® system that you can use to visually analyze the performance of web applications, TCP traffic, DNS traffic, FastL4, and overall system statistics. The statistics are displayed in graphical charts where you can drill down into a specific time range or system aspect to better understand network performance on certain devices, IP addresses, memory and CPU utilization, and so on. You can further focus the statistics in the charts by selecting dimension entities such as applications or virtual servers. About HTTP Analytics profiles An HTTP Analytics profile ( Local Traffic > Profiles > Analytics > HTTP Analytics ) is a set of definitions that determines the circumstances under which the system gathers, logs, notifies, and graphically displays information regarding traffic to an application. Each monitored application is associated with an HTTP Analytics profile. You associate the HTTP Analytics profile with one or more virtual servers used by the application. Each virtual server can have one HTTP and/or one TCP Analytics profile associated with it. Required License and Provisioning BIG-IP AVR module should be licensed and provisioned for AVR features to be available. Here is a sample screenshot of data gathered from a Virtual Server with HTTP Analytics profile applied. This can be viewed from the Configuration Utility Statistics››Analytics : HTTP : Overview We can clearly see the HTTP data gathered are common HTTP statistics that we would like to observe and have an understanding during a HTTP Brute force attack, Source Client IP address, Countries, User Agents, URLs and more and its corresponding statistics such as Average Transaction per second, Average Server Latency (ms) and more, are helpful information to help deciding what possible mitigation will be useful to quickly mitigate the attack. In the sample screenshot, we can see the User Agents statistics that "curl/7.47.1" sent bulk of the request to the Virtual Server. For this sample scenario, If this client HTTP User Agent is not expected and the HTTP application expects only real web browsers, the User Agent can be considered to be blocked from accessing the Virtual Server. Similarly, specific IP addresses or Countries may be unexpected traffic sources and are sending high volume traffic, these can also be considered to be blocked depending on what is considered expected traffic. These AVR statistics can also be exported in PDF format for offline review. HTTP Analytics profile can created from the Configuration Utility, Local Traffic››Profiles : Analytics : HTTP Analytics Here is a sample screenshot of a custom HTTP Analytics profile and is applied to a Virtual Server. Additional Notes for AVR HTTP request data gatheringfor HTTP Brute Force attack Public facing HTTP application are expected to receive malicious traffic. ProvisioningAVR module, configuring a HTTP Analytics profile and applying it to a HTTP Virtual Server in advance before a HTTP Brute Force attack happens provides advantages. It allows the gathering of HTTP traffic related statistics to understand the HTTP traffic pattern for a particular Virtual Server and helps in defining a traffic baseline and what normal traffic looks like and several other characteristic of traffic such as source IP its geolocation/country information, User Agent information and more. As an example of howAVR HTTP request data gatheredmay be used for HTTP Brute Force attack prevention, an application's login URL may be receiving excessive requestsas observed in Configuration Utility Statistics››Analytics : HTTP : Overview for a HTTP based Virtual Server with the HTTP analytics applied - in AVR, URL statistics can be reviewed and "Ave TPS" and "Transactions" are available - and application owner may define this an attack. Without HTTP Analytics profile applied to a Virtual Server, alternative manual methods or monitoring system would need to have been in place to describe the suspected HTTP Brute Force attack event.HTTP traffic information, very common, the User Agent may be a spoofed and upon reviewing the amount of transaction it sent during the HTTP Brute Force attack event, application owner may decide to block the specific User Agent value. Sample Application of AVR and LTM policy Data gathering techniques In this sample HTTP Analytics screenshot, the login URL /WackoPicko/users/login.php received high volume transaction from a specific source IP address. A closer look at the page information, an unusual looking User Agent was sending the suspicious amount of traffic. An application owner may consider this as an attack to the login URL and have the specific IP address or the User Agent to be blocked. Applying the sample LTM Policy where HTTP request data was logged, in reference to the HTTP AVR data screenshot,it can be observed that a HTTP Brute Force Attack was happening.username=!@#$%25 and random password are being sent to the login URL /WackoPicko/users/login.php. sample logs of a HTTP Brute Force Attack captured using thesample LTM Policy [root@BIG-IP:Active:Standalone] config # grep username /var/log/ltm | head Nov 30 17:54:10 BIG-IP info tmm4[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49133 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=12345678 Nov 30 17:54:10 BIG-IP info tmm5[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49134 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=jessica Nov 30 17:54:10 BIG-IP info tmm6[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49135 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=lovely Nov 30 17:54:10 BIG-IP info tmm4[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49136 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=abc123 Nov 30 17:54:10 BIG-IP info tmm5[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49137 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=monkey Nov 30 17:54:10 BIG-IP info tmm6[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49138 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=123456789 Nov 30 17:54:10 BIG-IP info tmm7[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49139 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=654321 Nov 30 17:54:10 BIG-IP info tmm[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49140 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=12345 Nov 30 17:54:10 BIG-IP info tmm1[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49141 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=daniel Nov 30 17:54:10 BIG-IP info tmm2[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49142 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=princess Using the sample generated logs during the HTTP Brute Force attack, running a combination of grep/cut/awk/uniq/sort command provides insight the top 10 username's approximate number login attempts grep/cut/awk/uniq/sort sample command: grep username /var/log/ltm | cut -d " " -f 27 | awk -F "&" '{print $1}' | awk -F "=" '{print $2}' | uniq -c | sort -rnk 1 | head sample output: [root@sec8:Active:Standalone] config # grep username /var/log/ltm | cut -d " " -f 27 | awk -F "&" '{print $1}' | awk -F "=" '{print $2}' | uniq -c | sort -rnk 1 | head 1006 aberrational 1006 abe 1006 abdul 1006 abduct 1006 abdominally 1006 abdominal 1006 abdomen 1006 abdication 1006 abcdefg 1006 abcdef Similarly, looking specifically for a username's approximate number of login attempts- in this example, '!@#$%25' sample command: grep '!@#$%25' /var/log/ltm | wc -l sample output: [root@sec8:Active:Standalone] config # grep '!@#$%25' /var/log/ltm | wc -l 4024
Most efficient methods for Connection logging?
Does anyone have real world experience with logging connections at a high rate? If so, which methods are you using to collect and transmit the data? We have a requirement to log all connections going through our F5 devices. Things like the client/server-side IPs/ports as well as HTTP details for HTTP VIPs and DNS details from our GTMs. It's the Whitehouse M-21-31 mandate if anyone if familiar with it. I've used Request Logging Profiles and various iRules with HSL to collect this type of data before, but I've never been too concerned about overhead because I would only apply them as needed, like when t-shooting an issue with a VIP. Our busiest appliance pushes around 150k conn/sec and 5k HTTP req/sec, so I now have consider the most efficient methods to avoid any kind of impact to traffic flows. I've done some lab testing with several different methods but I can't do any meaningful load tests in that environment.Below are some of my opinions based on my lab testing so far. Data Collection AVR - I like that this single feature can meet all the requirements for collecting TCP, HTTP, and DNS data. It would also be relatively easy to perform audits to ensure the VIPs have the necessary Analytics profiles as we can manage it from the AVR profiles themselves. My main concern is the overhead that results from the traffic analysis. I assume it has to maintain a large database where it stores all the analyzed data even if we just ship it off to Splunk. Even the data shipped off to Splunk includes several different logs for each connection (each with a different 'Entity'). Request Logging Profile- This is fairly flexible and should have low overhead since the F5 doesn't need to analyze any of the data like AVR does. This only collects HTTP data so we still need another solution to collect details for non HTTP VIPs. It would be a pain to audit since we don't have use any kind of deployment templates or automation. iRule - This provides a lot of flexibility and it is capable of collecting all the necessary data, but I don't know how well performance overhead compares to AVR. This would also be a pain to audit due to lack of deployment templates and automation. Data Transmission HSL UDP Syslog- I imagine this is the most efficient method to send events, but it's likely only a matter of time before we are required to use TCP/TLS. Telemetry Streaming - This is the more modern method and it offers some interesting features like System Poller, which could eventually allow us to move away from SNMP polling. We would need a workaround for our GTM-only devices because they cannot run a TS listener.
AVR average server latency istats with Irule
I am looking for a particular use case to be able to read the AVR analytics average server response time and be able to mark the specific server down and use this information to make a proper load balancing decision. So basically lets say when the average server response latency is higher then 1000ms then we should not send traffic there but do a load balancing reselection. The following command is used in TMSH to view this data. Now I would like to collect this within the Irule. show analytics http report view-by pool-member range now-3d measures {average-server-latency } In the analytics profile I have seen the option "publish Irule statistics" and that is what I need, but how do I read the specific metric data from within the Irule? I read about istats is that what I need?
Python module to post and retrieve IControl Rest JSON objects for AVR statistics
Problem this snippet solves: This module simplifies making Python dictionary objects that are converted to IControl rest AVR JSON objects. It also handles making AVR requests and retrieving results as well allowing multiple AVR requests to be queued, posted and retrieved. It also has some basis type checking for the elements of a AVR request. This module requires Bigip 12.1 on the target that statistics are retrieved. How to use this snippet: The main class is rest_avr.avr_req. It is a dictionary class that maps directly to an IControl Rest AVR JSON request as translated by json.dumps. Each dictionary element is an object derived from a customer class for each part of the request. The element classes have add() and clear() functions. if the element class only allows one entry the add() function will replace the existing entry, otherwise it will append the entry to the request element. The rest_avr.avr_req class also has functions to populate the HTTP host and authentication values for the target system. rest_avr.avr_req.post_and_response returns the Python representation of the JSON result of the query. rest_avr.avr_req.add_to_queue() adds the currently constructed request to a queue of requests to post. rest_avr.avr_req.post_and_response_queue() returns a python list of results of queued queries. The following code sample constructs, posts and returns results for an AVR statistics request for specific DNS records and a specificrecord type, then queues multiple quests and posts and returns results. #!/usr/bin/python import json import sys import time import rest_avr #print rest_avr.ShowAVRJsonApi #Populate the url avr_dns_req=rest_avr.avr_req() avr_dns_req.auth('admin','admin') avr_dns_req.url_base('10.10.2.113','dns') #Populate the json object avr_dns_req['analyticsModule'].add('dns') avr_dns_req['reportFeatures'].add('time-aggregated') avr_dns_req['entityFilters'].add('domain-name', 'OPERATOR_TYPE_EQUAL', ['test2.test1.com','test1.test1.com']) avr_dns_req['entityFilters'].add('query-type', 'OPERATOR_TYPE_EQUAL', ['a']) avr_dns_req['viewMetrics'].add('packets') avr_dns_req['viewDimensions'].add('domain-name') avr_dns_req['metricFilters'].add('packets', 'OPERATOR_TYPE_GREATER_THAN', 0) avr_dns_req['sortByMetrics'].add('packets', 'ascending') avr_dns_req['pagination'].add(20, 0) avr_dns_req['timeRange'].add(1461778251000000, None) #Post and retrieve results. result_py=avr_dns_req.post_and_response() if result_py != None: print ('\n' + result_py['results']['timeAggregated'][0]['dimensions'][0]['value'] + " " + result_py['results']['timeAggregated'][0]['metricValues'][0]['value'] + '\n') else: print result_py.error_layer print result_py.error_code print result_py.error_text # Now add multiple requests to a queue avr_dns_req.add_to_queue() avr_dns_req['entityFilters'].clear() avr_dns_req['entityFilters'].add('query-type', 'OPERATOR_TYPE_EQUAL', ['aaaa']) avr_dns_req.add_to_queue() #post and retrieve queued results result_py_q=avr_dns_req.post_and_response_queue() for result_py in result_py_q: if result_py != None: print ('\n' + result_py['results']['timeAggregated'][0]['dimensions'][0]['value'] + " " + result_py['results']['timeAggregated'][0]['metricValues'][0]['value'] + '\n') else: print result_py.error_layer print result_py.error_code print result_py.error_text Code : """ rest_avr provides a python interface to Bigip AVR statistics using the REST API. The main Python rest_avr.avr_req object is a Python dictionary that maps to a JSON object that can be processed with the json.dumps() function An IControl Rest AVR JSON request and response can be initiated with avr_req.post_and_response The simple description of the API can is available at avr_req.ShowJsonApi() Each of these modules has a method to add single or multiple elements as appropriate to the specific module. Once these elements are are populated a RestAPI request can be made with results returned as a python representation. avr_req.auth(user, passw) avr_req.url_base(host, module) avr_req['analyticsModule'].add(module) avr_req['analyticsModule'].clear() avr_req['reportFeatures'].add(metric_name, predicate, value) avr_req['reportFeatures'].clear() avr_req['entityFilters'].add(dimension_name, predicate, values) avr_req['entityFilters'].clear() avr_req['viewMetrics'].add(metric_name) avr_req['viewMetrics'].clear() avr_req['viewDimensions'].add(metric_name, order) avr_req['viewDimensions'].clear() avr_req['metricFilters'].add(metric_name, predicate, valu) avr_req['metricFilters'].clear() avr_req['sortByMetrics'].add(metric_name, orde) avr_req['sortByMetrics'].clear() avr_req['pagination'].add(num_results, skip_result) avr_req['pagination'].clear() avr_req['timeRange'].add(t_from, t_to) avr_req['timeRange'].clear() After a request in constructed a REST API call is initiated with initiated with: avr_req.post_and_response() The response is a python dictionary data structure of the results as processed by json.loads """ from copy import deepcopy import requests import json import sys import time import warnings __author__ = 'Mark Lloyd' __version__ = '1.0' # 05/24/2016 import json import requests import time class BadDictElement(Exception): def __init__(self, key, value, expl): Exception.__init__(self, '{0} {1} {2} '.format(key, value, expl)) class BadTime(Exception): def __init__(self, variable, value): Exception.__init__(self, '{0} {1} should be 16 char decimal in microseconds '.format('a', 'b')) class RequestFailure(Exception): def __init__(self, key, value): Exception.__init__(self, '{0} {1} '.format(key, value)) class analyticsModule(str): """ This class is tied to the structure of the parent class. parent() get's the parent object so we can make the string pseudo-mutable. accessed from within an avr request ['analyticsModule'].add(module) Adds a single string to analyticsModule element . If one exists it is replaced. ['analyticsModule'].clear() Send a null value to the analyticsModule element. See rest_avr.ShowAVRJsonApi for more details """ def parent(self, parent): self.parent = parent def add(self, module): """ avr_req.['analyticsModule'].add(module) Adds a single string to analyticsModule element . If one already exists it is replaced. This should be the same as the module string in avr_req.url_base. """ self.parent['analyticsModule'] = analyticsModule(module) self.parent['analyticsModule'].parent = self.parent def clear(self): """ avr_req.['analyticsModule'].add(module) replaces the analyticsModule mddule with a null string """ self.parent['analyticsModule'] = analyticsModule('') self.parent['analyticsModule'].parent = self.parent class metricFilters(list): """ avr_req.['metricFilters'].add(metric_name, predicate, value) metric name is a string, value is an integer Valid predicates strings are ['OPERATOR_TYPE_EQUAL', 'OPERATOR_TYPE_NOT_EQUAL', 'OPERATOR_TYPE_GREATER_THAN', OPERATOR_TYPE_LOWER_THAN','OPERATOR_TYPE_GREATER_THAN_OR_EQUAL', 'OPERATOR_TYPE_LOWER_THAN_OR_EQUAL']) avr_req['metricFilters'].clear() Clears metricFilters elements See rest_avr.ShowAVRJsonApi for more details. """ def __init__(self): self.append([]) self.valid_metric_predicate = ( ['OPERATOR_TYPE_EQUAL', 'OPERATOR_TYPE_NOT_EQUAL', 'OPERATOR_TYPE_GREATER_THAN', 'OPERATOR_TYPE_LOWER_THAN', 'OPERATOR_TYPE_GREATER_THAN_OR_EQUAL', 'OPERATOR_TYPE_LOWER_THAN_OR_EQUAL']) def add(self, metric_name, predicate, value): """ avr_req.['metricFilters'].add(metric_name, predicate, value) metric name is a string, value is an integer Valid predicates strings are ['OPERATOR_TYPE_EQUAL', 'OPERATOR_TYPE_NOT_EQUAL', 'OPERATOR_TYPE_GREATER_THAN', OPERATOR_TYPE_LOWER_THAN','OPERATOR_TYPE_GREATER_THAN_OR_EQUAL', 'OPERATOR_TYPE_LOWER_THAN_OR_EQUAL'] """ if type(value) is not int: raise BadDictElement(metric_name, value, 'value should be integer') if predicate in self.valid_metric_predicate: # first check if it is already there for metric in self[0]: if metric['metricName'] == metric_name: metric['predicate'] = predicate metric['value'] = value return 0 # if it is not there then just add it. self[0].append({'metricName': metric_name, 'predicate': predicate, 'value': value}) else: raise BadDictElement(metric_name, predicate, 'invalid predicate') def clear(self): """ avr_req['metricFilters'].clear() Clears metricFilters elements """ del self[0][:] class entityFilters(list): """ avr_req.['entityFilters'].add(dimension_name, predicate, values): All values are strings valid predicate is 'OPERATOR_TYPE_EQUAL' ['entityFilters'].clear() Clears the entityFilters element See rest_avr.ShowJsonApi for more details """ def __init__(self): self.append([]) def add(self, dimension_name, predicate, values): """ avr_req.['entityFilters'].add(dimension_name, predicate, values): All values are strings valid predicate is 'OPERATOR_TYPE_EQUAL' """ if predicate is 'OPERATOR_TYPE_EQUAL': # then loop throuth to see if the dimenson name already exists, if so replace for entity in self[0]: if entity['dimensionName'] == dimension_name: entity['predicate'] = predicate entity['values'] = values return 0 # if it is not there then just add it. self[0].append({'dimensionName': dimension_name, 'predicate': predicate, 'values': values}) else: raise BadDictElement(dimension_name, predicate, 'predicate must be OPERATOR_TYPE_EQUAL') def clear(self): """ ['entityFilters'].clear() Clears the entityFilters element """ del self[0][:] class reportFeatures(list): """ avr_req.['reportFeatures'].add( feature) adds report feature string. Multiple features are permitted. ['reportFeatures'].clear() Clears the analyticsModule element. See rest_avr.ShowAVRJsonApi for more details. """ def add(self, feature): """ avr_req.['reportFeatures'].add( feature) adds report feature string. Multiple features are permitted .""" if feature not in self: self.append(feature) def clear(self): """ ['reportFeatures'].clear() Clears the entityFilters element """ del self[:] class sortByMetrics(list): """ avr_req.['sortByMetrics'].add(metric_name, order) valid order names are 'ascending' and 'descending' sortByMetrics is optional in an AVR request. avr_req['sortByMetrics'].clear() Clears the sortByMetrics element. See rest_avr.ShowAVRJsonApi for more details. """ def __init__(self): self.metric_list = [] def add(self, metric_name, order): if metric_name not in self.metric_list: self.append({'metricName': metric_name, 'order': order}) self.metric_list.append(metric_name) def clear(self): """ ['sortByMetrics'].clear() Clears the sortByMetrics element """ del self[:] del self.metric_list[:] class viewDimensions(list): """ avr_req.['viewDimensions'].add(dimension_name): adds view dimension, only one dimension is allowed add will replace element if it already exists avr_req['viewDimensions'].clear() Clears the viewDimensions element. See rest_avr.ShowAVRJsonApi for more details. """ def __init__(self): self.append([]) self[0] = {} def add(self, dimension_name): """ avr_req.['viewDimensions'].add(dimension_name): adds view dimension string, only one dimension is allowed add will replace element if it already exists """ self[0]['dimensionName'] = dimension_name def clear(self, dimension_name): """ ['viewDimensions'].clear() Clears the viewDimensions element """ del self[0][:] class viewMetrics(list): """ avr_req.['viewMetrics'].add(metric_name): appends metric_name string to list. The specification allows multiple view metric elements avr_req['viewMetrics'].clear() Clears the viewMetrics elements See rest_avr.ShowAVRJsonApi for more details. """ def __init__(self): self.metric_list = [] def add(self, metric_name): """ avr_req.['viewMetrics'].add(metric_name): appends metric_name string to list. The specification allows multiple viewMetric elements """ if metric_name not in self.metric_list: self.append({'metricName': metric_name}) self.metric_list.append(metric_name) def clear(self): """ ['viewMetrics'].clear() Clears the viewMetrics elements """ del self[:] del self.metric_list[:] class timeRange(dict): """ avr_req.['timeRange'].add( t_from, t_to) both values are 16 digit numeric value in microseconds of unix/linux time. t_to is optional and can be replace by None timeRange is an optional. avr_req['timeRange'].clear() Clears the timeRange elements See rest_avr.ShowAVRJsonApi for more details. """ def add(self, t_from, t_to): """ avr_req.['timeRange'].add( t_from, t_to) both values are 16 digit numeric value in microseconds of unix/linux time. t_to is optional and can be replace by None timeRange is optional. """ if type(t_from) is long and len(str(t_from)) == 16: self['from'] = t_from else: raise BadTime(t_from + " is 16 digit numeric value in microseconds") if t_to != '' and t_to != 0 and t_to != None: if type(t_to) is long and len(str(t_from)) == 16: self['to'] = t_to else: raise BadTime(t_to + " is 16 digit numeric value in microseconds") else: if 'to' in self.keys(): del self['to'] def clear(self): """ ['timeRange'].clear() Clears the timeRange element """ del self[:] class pagination(dict): """ avr_req.['pagination'].add(num_results, skip_results) both are integer values. avr_req['pagination'].clear() Clears the pagination elements See rest_avr.ShowAVRJsonApi for more details. """ def add(self, num_results, skip_results): """ avr_req.['pagination'].add(num_results, skip_results) both arguments are integers. """ if type(num_results) is int: self['numberOfResults'] = num_results else: raise BadDictElement('number of Results ', num_results, 'must be integer') if type(skip_results) is int: self['skipResults'] = skip_results else: raise BadDictElement('skipResults ', skip_results, 'must be integer') def clear(self): """ ['pagination'].clear() Clears the pagination element """ del self[:] class avr_resp(dict): """ python response error is applicable. """ def __init__(self): self.error_layer = None self.error_code = None self.error_text = None class avr_req(dict): """ The main class for rest_avr. avr_req contains a dictionary that maps to the elements of a Icontrol REST AVR request along with capability of posting that request and receiving a response. The dictionary values are object instances of python classes that correspond to the the JSON values of the object's name/value pair. Each value has two public methods: avr_req.['objectName']add(): adds an element to the appropriate object with type checking. If an element allows more then one instance the add function will append the element If an element allows only one instance the add function will replace the element avr_req.['objectName'].clear()r: clears all elements in the object. printing rest_avr.ShowAVRJsonApi provides documentation for the AVR JASON elements. Further documentation is available on devcentral.f5.com To post an AVR Rest request there are two functions to populate the HTTP/HTTPS request. avr_req.auth(user, passw): provides the username and password avr_req.url_base(host, module) provides the host and the bigip module AVR queries to construct the URL to make the request. Then to post the request and return results in a python representation of the JSON response. avr_req.post_and_response() """ def __init__(self): self['analyticsModule'] = analyticsModule() self['analyticsModule'].parent = self self['pagination'] = pagination() self['metricFilters'] = metricFilters() self['entityFilters'] = entityFilters() self['reportFeatures'] = reportFeatures() self['sortByMetrics'] = sortByMetrics() self['viewDimensions'] = viewDimensions() self['viewMetrics'] = viewMetrics() self['timeRange'] = timeRange() self.avr_session = requests.session() self.avr_session.verify = False self.avr_session.headers.update({'Content-Type': 'application/json'}) # for multiple queued request handling. self.req_queue = [] self.generate_id = None self.done = None self.result = None self.num_requests = 0 self.res_queue = [] def post_and_response(self): """ returns a python representation of the json response to the request. failure returns array ['ERROR','component',error] """ warnings.filterwarnings("ignore") self.generate_request = self.avr_session.post(self.req_url_base + "/generate-report/", data=json.dumps(self)) self.generate_request_py = json.loads(self.generate_request.text) self.result_guid = self.generate_request_py['id'] self.results_status_url = self.req_url_base + "/generate-report/" + self.result_guid + "/?$select=status,reportResultsLink" self.results_url = self.req_url_base + "/report-results/" + self.result_guid self.sleeptime = .5 for i in range(5): time.sleep(self.sleeptime) self.sleeptime *= 2 # double backoff period each time. self.status_results_json = self.avr_session.get(self.results_status_url) self.status_results = json.loads(self.status_results_json.text) if self.status_results['status'] == 'FAILED': self.result = avr_resp() self.result_error_layer = 'REST' self.result_error_code = self.status_results['status'] self.result.error_text = self.status_results if self.status_results['status'] == 'FINISHED': self.raw_results_url = self.status_results['reportResultsLink'] self.results_url = self.raw_results_url.replace('localhost', self.host_name) self.results = self.avr_session.get(self.results_url) if self.results.status_code == 200: self.result = avr_resp() self.result.update(json.loads(self.results.text)) return self.result else: self.result = avr_resp() self.result.error_layer = 'HTTP' self.result.error_code = self.results.status_code self.result.error_text = self.results return self.result else: continue self.result = avr_resp() self.result.error_layer = 'REST_AVR' self.result.error_code = '408' self.result.error_text = 'TIMEOUT' def auth(self, user, passw): """ avr_req.auth(user, passw): username and password """ self.avr_session.auth = (user, passw) def url_base(self, host, module): """ avr_req.url_base(host, module) host and bigip module AVR queries to construct the URL to make the request. """ self.host_name = host self.req_url_base = 'https://%s/mgmt/tm/analytics/%s' % (host, module) self.module_py = {'analyticsModule': module} def add_to_queue(self): "adds request as currently constructed to queue" self.req_queue.append(deepcopy(self)) def clear_queue(self): """" clears request queue """ del self.req_queue[:] def post_and_response_queue(self): """ posts and sends response to from queue of requests. """ warnings.filterwarnings("ignore") for req in self.req_queue: req.generate_request = req.avr_session.post(req.req_url_base + "/generate-report/", data=json.dumps(req)) req.generate_request_py = json.loads(req.generate_request.text) req.generate_id = (req.generate_request_py['id']) req.results_status_url = self.req_url_base + "/generate-report/" + req.generate_id + "/?$select=status,reportResultsLink" self.sleeptime = .5 self.num_requests = len(self.req_queue) for i in range(5): for req in self.req_queue: if req.done is None: time.sleep(self.sleeptime) self.sleeptime *= 2 # double backoff period each time. req.status_results_json = req.avr_session.get(req.results_status_url) req.status_results = json.loads(req.status_results_json.text) if req.status_results['status'] == 'FAILED': req.result = avr_resp() req.result_error['layer'] = 'REST' req.result_error['error'] = req.status_results['status'] req.result_error['text'] = req.status_results if req.status_results['status'] == 'FINISHED': req.raw_results_url = req.status_results['reportResultsLink'] req.results_url = req.raw_results_url.replace('localhost', self.host_name) req.results = self.avr_session.get(req.results_url) if req.results.status_code == 200: req.result = avr_resp() req.result.update(json.loads(req.results.text)) req.done = True self.res_queue.append(req.result) self.num_requests -= 1 else: req.result = avr_resp() req.result_error.layer = 'HTTP' req.result_error.code = req.results.status_code req.result_error.text = req.results self.res_queue.append(req.result) if i == 5: if req.result == False: req.result = avr_resp() req.result.error_layer = 'REST_AVR' req.result.error_error = '408' req.result.error_text = 'TIMEOUT' if self.num_requests == 0: break return self.res_queue ShowAVRJsonApi = """ reportFeatures -------------- Specifies the kind of information that appears in a response from AVR. You may specify one or more of the following values: existing-entities time-aggregated time-series entities-count viewDimensions -------------- Specifies the dimensions for which to calculate a report, such as: {"dimensionName": "domain-name"} You may only specify a single dimension. You may omit this field in a report generation request. viewMetrics ----------- Specifies the list of metrics by which to sort results, such as: { "metricName": "average-tps" }, { "metricName": "transactions" } If you specify either time-aggregated or time-series features, you must specify one metric in a report generation request. sortByMetrics -------------- Specifies the list of metrics to sort by, such as: [{ metricName: "average-tps", order:"descending" } ] Valid values are ascending and descending. Sorting only applies to the time-aggregated feature. You do not need to specify this field in a report generation request. timeRange --------- Specifies the time range, in microseconds, for which to calculate a report, such as: {"from": 1410420888000000, "to": 1410424488000000 } You do not need to specify this field in a report generation request. entityFilters ============= Specifies the entities and values for which to calculate a report. You can specify a single entity with a second level of dimension filters that describe an aspect of the entity. If you specify multiple entity types, the results include only the entities that match all of the criteria. You do not need to specify this field in a report generation request. The following snippet contains two entities with corresponding values: [[{ "dimensionName" : "virtual", "predicate": "OPERATOR_TYPE_EQUAL", "values : ["phpAuction_VS_1"] }, { "dimensionName : "response-code", "predicate": "OPERATOR_TYPE_EQUAL", "values" : ["200"] } ]] metricFilters ------------- Specifies the metric filters for which to calculate a report, such as: [{ "metricName": "transactions", "predicate" : metricFilters "OPERATOR_TYPE_GREATER_THAN" "value": 100 }] You do not need to specify this field in a report generation request. For the existing-entities feature, AVR supports the OPERATOR_TYPE_LIKE predicate. AVR also supports the following predicates: OPERATOR_TYPE_EQUAL OPERATOR_TYPE_NOT_EQUAL OPERATOR_TYPE_GREATER_THAN OPERATOR_TYPE_LOWER_THAN OPERATOR_TYPE_GREATER_THAN_OR_EQUAL OPERATOR_TYPE_LOWER_THAN_OR_EQUAL pagination ---------- Specifies the number of results to return, and the number of results to skip, such as: { numberOfResults : 10, skipResults : 10} To see the second set of ten results, use the example shown here. AVR does not implement the OData query parameters top or skip. In order to see a specific set of results, you must set the number of results to return and then determine how many results to skip. You do not need to specify this field in a report generation request. """ Tested this on version: 12.0
AVR Profile and Remote syslog
Hi all, Just wondering if anyone has managed to use an analytics profile to export the logs/data to an external source, ideally rsyslog? There was a lot of reading about SPLUNK and iApp's and also setting up log publishers/destinations, but I could not find anyone having success logging to a remote syslog server direct from an analytics profile. I think this is possible from the manual of AVR. Any help would be great, Thanks.AVR remote logging HSL pool
Dear All, I am trying to set up AVR remote logging to send web statistics to an external syslog server, but no traffic is being send at all. I configured the pool for sending syslog traffic. Configured ARV Analytics profile for external statistics logging type enabled. The log publisher named syslog used in the Analytics profile refers to the Syslog-ECM log destination. The syslog-ECM log destination is configured as type syslog and refers to the syslog pool. But traffic is not being send. If I use an Irule for sending logs to the destination it does work, it is not an IP connectivity issue but a more AVR specific one. Someone tried this out before, please leave your comments. Thanks
