Implementing BIG-IP WAF logging and visibility with ELK
Scope This technical article is useful for BIG-IP users familiar with web application security and the implementation and use of the Elastic Stack.This includes, application security professionals, infrastructure management operators and SecDevOps/DevSecOps practitioners. The focus is for WAF logs exclusively.Firewall, Bot, or DoS mitigation logging into the Elastic Stack is the subject of a future article. Introduction This article focusses on the required configuration for sending Web Application Firewall (WAF) logs from the BIG-IP Advanced WAF (or BIG-IP ASM) module to an Elastic Stack (a.k.a. Elasticsearch-Logstash-Kibana or ELK). First, this article goes over the configuration of BIG-IP.It is configured with a security policy and a logging profile attached to the virtual server that is being protected. This can be configured via the BIG-IP user interface (TMUI) or through the BIG-IP declarative interface (AS3). The configuration of the Elastic Strack is discussed next.The configuration of filters adapted to processing BIP-IP WAF logs. Finally, the article provides some initial guidance to the metrics that can be taken into consideration for visibility.It discusses the use of dashboards and provides some recommendations with regards to the potentially useful visualizations. Pre-requisites and Initial Premise For the purposes of this article and to follow the steps outlined below, the user will need to have at least one BIG-IP Adv. WAF running TMOS version 15.1 or above (note that this may work with previous version but has not been tested).The target BIG-IP is already configured with: A virtual Server A WAF policy An operational Elastic Stack is also required. The administrator will need to have configuration and administrative privileges on both the BIG-IP and Elastic Stack infrastructure.They will also need to be familiar with the network topology linking the BIG-IP with the Elastic Search cluster/infrastructure. It is assumed that you want to use your Elastic Search (ELK) logging infrastructure to gain visibility into BIG-IP WAF events. Logging Profile Configuration An essential part of getting WAF logs to the proper destination(s) is the Logging Profile.The following will go over the configuration of the Logging Profile that sends data to the Elastic Stack. Overview of the steps: Create Logging Profile Associate Logging Profile with the Virtual Server After following the procedure below On the wire, logs lines sent from the BIG-IP are comma separated value pairs that look something like the sample below: Aug 25 03:07:19 localhost.localdomainASM:unit_hostname="bigip1",management_ip_address="192.168.41.200",management_ip_address_2="N/A",http_class_name="/Common/log_to_elk_policy",web_application_name="/Common/log_to_elk_policy",policy_name="/Common/log_to_elk_policy",policy_apply_date="2020-08-10 06:50:39",violations="HTTP protocol compliance failed",support_id="5666478231990524056",request_status="blocked",response_code="0",ip_client="10.43.0.86",route_domain="0",method="GET",protocol="HTTP",query_string="name='",x_forwarded_for_header_value="N/A",sig_ids="N/A",sig_names="N/A",date_time="2020-08-25 03:07:19",severity="Error",attack_type="Non-browser Client,HTTP Parser Attack",geo_location="N/A",ip_address_intelligence="N/A",username="N/A",session_id="0",src_port="39348",dest_port="80",dest_ip="10.43.0.201",sub_violations="HTTP protocol compliance failed:Bad HTTP version",virus_name="N/A",violation_rating="5",websocket_direction="N/A",websocket_message_type="N/A",device_id="N/A",staged_sig_ids="",staged_sig_names="",threat_campaign_names="N/A",staged_threat_campaign_names="N/A",blocking_exception_reason="N/A",captcha_result="not_received",microservice="N/A",tap_event_id="N/A",tap_vid="N/A",vs_name="/Common/adv_waf_vs",sig_cves="N/A",staged_sig_cves="N/A",uri="/random",fragment="",request="GET /random?name=' or 1 = 1' HTTP/1.1\r\n",response="Response logging disabled" Please choose one of the methods below.The configuration can be done through the web-based user interface (TMUI), the command line interface (TMSH), directly with a declarative AS3 REST API call, or with the BIG-IP native REST API.This last option is not discussed herein. TMUI Steps: Create Profile Connect to the BIG-IP web UI and login with administrative rights Navigate to Security >> Event Logs >> Logging Profiles Select “Create” Fill out the configuration fields as follows: Profile Name (mandatory) Enable Application Security Set Storage Destination to Remote Storage Set Logging Format to Key-Value Pairs (Splunk) In the Server Addresses field, enter an IP Address and Port then click on Add as shown below: Click on Create Add Logging Profile to virtual server with the policy Select target virtual server and click on the Security tab (Local Traffic >> Virtual Servers : Virtual Server List >> [target virtualserver] ) Highlight the Log Profile from the Available column and put it in the Selected column as shown in the example below (log profile is “log_all_to_elk”): Click on Update At this time the BIG-IP will forward logs Elastic Stack. TMSH Steps: Create profile ssh into the BIG-IP command line interface (CLI) from the tmsh prompt enter the following: create security log profile [name_of_profile] application add { [name_of_profile] { logger-type remote remote-storage splunk servers add { [IP_address_for_ELK]:[TCP_Port_for_ELK] { } } } } For example: create security log profile dc_show_creation_elk application add { dc_show_creation_elk { logger-type remote remote-storage splunk servers add { 10.45.0.79:5244 { } } } } 3. ensure that the changes are saved: save sys config partitions all Add Logging Profile to virtual server with the policy 1.From the tmsh prompt (assuming you are still logged in) enter the following: modify ltm virtual [VS_name] security-log-profiles add { [name_of_profile] } For example: modify ltm virtual adv_waf_vs security-log-profiles add { dc_show_creation_elk } 2.ensure that the changes are saved: save sys config partitions all At this time the BIG-IP sends logs to the Elastic Stack. AS3 Application Services 3 (AS3) is a BIG-IP configuration API endpoint that allows the user to create an application from the ground up.For more information on F5’s AS3, refer to link. In order to attach a security policy to a virtual server, the AS3 declaration can either refer to a policy present on the BIG-IP or refer to a policy stored in XML format and available via HTTP to the BIG-IP (ref. link). The logging profile can be created and associated to the virtual server directly as part of the AS3 declaration. For more information on the creation of a WAF logging profile, refer to the documentation found here. The following is an example of a pa rt of an AS3 declaration that will create security log profile that can be used to log to Elastic Stack: "secLogRemote": { "class": "Security_Log_Profile", "application": { "localStorage": false, "maxEntryLength": "10k", "protocol": "tcp", "remoteStorage": "splunk", "reportAnomaliesEnabled": true, "servers": [ { "address": "10.45.0.79", "port": "5244" } ] } In the sample above, the ELK stack IP address is 10.45.0.79 and listens on port 5244 for BIG-IP WAF logs.Note that the log format used in this instance is “Splunk”.There are no declared filters and thus, only the illegal requests will get logged to the Elastic Stack.A sample AS3 declaration can be found here. ELK Configuration The Elastic Stack configuration consists of creating a new input on Logstash.This is achieved by adding an input/filter/ output configuration to the Logstash configuration file.Optionally, the Logstash administrator might want to create a separate pipeline – for more information, refer to this link. The following is a Logstash configuration known to work with WAF logs coming from BIG-IP: input { syslog { port => 5244 } } filter { grok { match => { "message" => [ "attack_type=\"%{DATA:attack_type}\"", ",blocking_exception_reason=\"%{DATA:blocking_exception_reason}\"", ",date_time=\"%{DATA:date_time}\"", ",dest_port=\"%{DATA:dest_port}\"", ",ip_client=\"%{DATA:ip_client}\"", ",is_truncated=\"%{DATA:is_truncated}\"", ",method=\"%{DATA:method}\"", ",policy_name=\"%{DATA:policy_name}\"", ",protocol=\"%{DATA:protocol}\"", ",request_status=\"%{DATA:request_status}\"", ",response_code=\"%{DATA:response_code}\"", ",severity=\"%{DATA:severity}\"", ",sig_cves=\"%{DATA:sig_cves}\"", ",sig_ids=\"%{DATA:sig_ids}\"", ",sig_names=\"%{DATA:sig_names}\"", ",sig_set_names=\"%{DATA:sig_set_names}\"", ",src_port=\"%{DATA:src_port}\"", ",sub_violations=\"%{DATA:sub_violations}\"", ",support_id=\"%{DATA:support_id}\"", "unit_hostname=\"%{DATA:unit_hostname}\"", ",uri=\"%{DATA:uri}\"", ",violation_rating=\"%{DATA:violation_rating}\"", ",vs_name=\"%{DATA:vs_name}\"", ",x_forwarded_for_header_value=\"%{DATA:x_forwarded_for_header_value}\"", ",outcome=\"%{DATA:outcome}\"", ",outcome_reason=\"%{DATA:outcome_reason}\"", ",violations=\"%{DATA:violations}\"", ",violation_details=\"%{DATA:violation_details}\"", ",request=\"%{DATA:request}\"" ] } break_on_match => false } mutate { split => { "attack_type" => "," } split => { "sig_ids" => "," } split => { "sig_names" => "," } split => { "sig_cves" => "," } split => { "staged_sig_ids" => "," } split => { "staged_sig_names" => "," } split => { "staged_sig_cves" => "," } split => { "sig_set_names" => "," } split => { "threat_campaign_names" => "," } split => { "staged_threat_campaign_names" => "," } split => { "violations" => "," } split => { "sub_violations" => "," } } if [x_forwarded_for_header_value] != "N/A" { mutate { add_field => { "source_host" => "%{x_forwarded_for_header_value}"}} } else { mutate { add_field => { "source_host" => "%{ip_client}"}} } geoip { source => "source_host" } } output { elasticsearch { hosts => ['localhost:9200'] index => "big_ip-waf-logs-%{+YYY.MM.dd}" } } After adding the configuration above to the Logstash parameters, you will need to restart the Logstash instance to take the new logs into configuration.The sample above is also available here. The Elastic Stack is now ready to process the incoming logs.You can start sending traffic to your policy and start seeing logs populating the Elastic Stack. If you are looking for a test tool to generate traffic to your Virtual Server, F5 provides a simpleWAF tester tool that can be found here. At this point, you can start creating dashboards on the Elastic Stack that will satisfy your operational needs with the following overall steps: ·Ensure that the log index is being created (Stack Management >> Index Management) ·Create a Kibana Index Pattern (Stack Management>>Index patterns) ·You can now peruse the logs from the Kibana discover menu (Discover) ·And start creating visualizations that will be included in your Dashboards (Dashboards >> Editing Simple WAF Dashboard) A complete Elastic Stack configuration can be found here – note that this can be used with both BIG-IP WAF and NGINX App Protect. Conclusion You can now leverage the widely available Elastic Stack to log and visualize BIG-IP WAF logs.From dashboard perspective it may be useful to track the following metrics: -Request Rate -Response codes -The distribution of requests in term of clean, blocked or alerted status -Identify the top talkers making requests -Track the top URL’s being accessed -Top violator source IP An example or the dashboard might look like the following:
Dashboard and logs show different bandwidth utilization?
I can't figure this one out, so I figure I'd ask the experts! I've found multiple times that the performance view or the dashboard on the F5 don't correlate to local traffic logs about bandwidth. I've seen this multiple times, but here is an example from yesterday with my 200Mbps LTM. The dashboard showed this (grabbed historic from performance statistics). At the same time, I was getting flooded with messages like this from my system local traffic log. Tue Jun 27 08:05:06 PDT 2017 notice PERF-F5 tmm[11064] 01010045 Bandwidth utilization is 244 Mbps, exceeded 75% of Licensed 200 Mbps I've looked at it multiple times and it just doesn't add up. I've had this issue in multiple environments. Can someone explain to me where I can actually see the bandwidth I'm using if it's not being represented in the dashboard? Thanks!Dos Attacks not showing on dashboard
Issue: DoS attacks are not showing on the DoS dashboard. Information: Strange part is this was working but then I made a few changes to split this particular virtual server into internal and external VIPs only apply DoS profile to external VIP. I have a DoS logging profile enabled on this VS I know attacks are happening as I can see them under Security > Event logs > DoS > Application Events However, any of the new attack ids don’t show under the DoS Dashboard located Security > Reporting > DoS > Dashboards Any thoughts on how to fix would be much appreciated? Thanks!MIB and/or OID of dashboard graph of throughput
Not having any luck finding the MIB for the throughput graph shown in the dashboard. I have all mibs downloaded and a snmpwalk of the host, but cannot find anything relevant that matches up. Can someone please point me in the right direction? BIG-IP VPR-C2400 11.4.1 Build 608.0 Final Closest previous post I could find here: https://devcentral.f5.com/questions/help-finding-oid-for-gtm-performance-graph
What is Exact Througput
Hi guys, I'm trying to figure out max throughput on LTM. So I've seen "Dashboard", "Performance Graph(Throughput)", "CLI Command(tmsh show sys performacne)". These values are all different. dashboard value changes in every 3 sec.( Performance graph(per 10sec), show sys perforamance throughput(8~15sec)) I'd like to know. What is real(?) Throughput? and Why there is difference?, What is difference? How LTM displays the throughput value. (polling OID internally? or something) If you know about this, HELP ME. Thank you, in advance.
Exposing F5 dashboard publicly?
Is it possible to expose F5 dashboard publicly (or at least with really minimal access to F5 Configuration Utility)? I'd like to expose it to machine displaying dashboards on display wall yet I wouldn't like to give it full access to actual F5 Configuration Utility interface. My question is either: How to expose dashboard without requiring authentication / requiring different authentication than F5 Configuration Utility or: Is there any lower permissions level for account in F5 than Guest that allows dashboard access or can this role permissions be even reduced to access only selected statistics required for dashboard operation? Another related topic would be: is it possible to expose dashboard on other vlan than management? For example as conventional Virtual Sever in some other network with destination pointing to F5 management IP?Dashboard red dots that say "Click to view config changes near this time"
Recently I noticed these red dots on the 11.4.1 dashboard that say "Click to view config changes near this time" when you hover over them. When you click a dot it gives you all sorts of useful info about the object that was modified/created/deleted, but it doesn't say who made the changes. I've searched the logs using the following commands but don't get anything useful: zcat /var/logs/audit.*.gz | grep zcat /var/logs/ltm.*.gz | grep All I get is an entry that says "user=root" saved the config and other entries that state the newly created objects were marked down. If only there was a place to look that links the username with the objects. Those red dots in the dashboard are awesome, but they miss that one critical piece of information (username). I'm sure if it were easy to do, or even possible, F5 would have done it. I guess I'm just venting. Anybody know any tricks?
Modified /Common/system mgmt_dhcp -- What is this?
Hi, I have a GTM that is a VIPRION guest. It runs 11.5.1 HF10. I noticed something strange in the Dashboard and I was wondering if anyone has encountered it before. Every Friday at around 7am, my GTM's throughput and connections drop by about 80%, then spike back up and continue on with a normal pattern. 2 weeks out of 4 there has been a Configuration Object Change. We haven't made any changes. The change is something I can't find any information about: Change Details - Modified /Common/system mgmt_dhcp instance-name /Common/system field_name mgmt_dhcp class-name system container (no value) config_source user No one has been complaining, and I just now noticed this. There's nothing in GLSB/Local Traffic/System logs around that time. We have another GTM vCMP guest running the same version, which shares config changes with this GTM. They aren't an HA pair though. The other GTM has the same symptoms, but it happened only 1 out of 4 weeks this month. Thanks, K
BIG-IQ DNS TPS Per Geo Location
Hi, I recently deployed a BIG-IQ, to manage all my F5 LTM and DNS Tenants, I'm reviewing the information shown on the different dashboards of BIG-IQ. On the DNS Dashboard, there is a section named TPS Per Geo Location. For some reason i'm just seeing the world map, but with no data. Does anyone knows how to enable information on this map? regards,
