F5 Analytics iApp
Problem this snippet solves: Analytics iApp v3.7.0 You can use this fully supported version of the analytics iApp template to marshal statistical and logging data from the BIG-IP system. The iApp takes this data and formats it as a JSON object which is then exported for consumption by data consumers, such as F5 BIG-IQ or applications such as Splunk. The Analytics iApp allows you to configure several categories of data to be exported. For data consumers like Splunk, the iApp lets you configure the network endpoint to which the data is sent. Version 3.7.0 of the iApp template is fully supported by F5 and available on downloads.f5.com. We recommend all users upgrade to this version. For more information, see https://support.f5.com/csp/article/K07859431. While this version of the iApp is nearly identical to the v3.6.13 which was available on this page, the major difference (other than being fully supported) is that ability to gather APM statistics using the iApp has been removed from BIG-IP versions prior to 12.0. Supported/Tested BIG-IP versions: 11.4.0 - 12.1.2. Data Sources: LTM, GTM, AFM, ASM, APM, SWG, and iHealth (APM statistics require 12.0 or later) Data Output Formats: Splunk, F5 Analytics, F5 Risk Engine Splunk App: https://apps.splunk.com/apps/id/f5 The new deployment guide can be found on F5.com: http://f5.com/pdf/deployment-guides/f5-analytics-dg.pdf Video Demo - https://player.vimeo.com/video/156773835 Solution Architecture - 20s Installation - 1m53s UI Demo Device Dashboard - 6m44s Application Issue Troubleshooting - 9m26s Application Team Self Service - 12m17s Code : https://downloads.f5.com/esd/ecc.sv?sw=BIG-IP&pro=iApp_Templates&ver=iApps&container=iApp-Templates
Adding the body of requests/responses to the data being logged to Splunk via iRule.
Hi All, We are presently using the iRule below to log request / response data to splunk. I'd like to add the body of the requests to our splunk logging. I had tried to user HTTP::payload as part of HTTP_REQUEST however it seems that the irule no longer functions when I place this there. When I add HTTP_REQUEST_DATA to the iRule to cater for HTTP:payload, I break the app - I expect that this is my implementation of HTTP_REQUEST_DATA. Is there an easy way to add the logging of the body of the request and response to what is sent to splunk? Thanks in advance when CLIENT_ACCEPTED { set client_address [IP::client_addr] set vip [IP::local_addr] } when HTTP_REQUEST { set http_host [HTTP::host]:[TCP::local_port] set http_uri [HTTP::uri] set http_url $http_host$http_uri set http_method [HTTP::method] set http_version [HTTP::version] set http_user_agent [HTTP::header "User-Agent"] set http_content_type [HTTP::header "Content-Type"] set http_referrer [HTTP::header "Referer"] set tcp_start_time [clock clicks -milliseconds] set req_start_time [clock format [clock seconds] -format "%Y/%m/%d %H:%M:%S"] set cookie [HTTP::cookie names] set user [HTTP::username] set virtual_server [LB::server] if { [HTTP::header Content-Length] > 0 } then { set req_length [HTTP::header "Content-Length"] } else { set req_length 0 } } when HTTP_RESPONSE { set res_start_time [clock format [clock seconds] -format "%Y/%m/%d %H:%M:%S"] set node [IP::server_addr] set node_port [TCP::server_port] set http_status [HTTP::status] set req_elapsed_time [expr {[clock clicks -milliseconds] - $tcp_start_time}] if { [HTTP::header Content-Length] > 0 } then { set res_length [HTTP::header "Content-Length"] } else { set res_length 0 } set hsl [HSL::open -proto TCP -pool p-remote-logging] HSL::send $hsl "<190>,f5_irule=Splunk-iRule-HTTP,src_ip=$client_address,vip=$vip,http_method=$http_method,http_host=$http_host,http_uri=$http_uri,http_url=$http_url,http_version=$http_version,http_user_agent=\"$http_user_agent\",http_content_type=$http_content_type,http_referrer=\"$http_referrer\",req_start_time=$req_start_time,cookie=\"$cookie\",user=$user,virtual_server=\"$virtual_server\",bytes_in=$req_length,res_start_time=$res_start_time,node=$node,node_port=$node_port,http_status=$http_status,req_elapsed_time=$req_elapsed_time,bytes_out=$res_length\r\n" } when LB_FAILED { set hsl [HSL::open -proto TCP -pool p-remote-logging] HSL::send $hsl "<190>,f5_irule=Splunk-iRule-LB_FAILED,src_ip=$client_address,vip=$vip,http_method=$http_method,http_host=$http_host,http_uri=$http_uri,http_url=$http_url,http_version=$http_version,http_user_agent=\"$http_user_agent\",http_content_type=$http_content_type,http_referrer=\"$http_referrer\",req_start_time=$req_start_time,cookie=\"$cookie\",user=$user,virtual_server=\"$virtual_server\",bytes_in=$req_length\r\n" }Getting Started with Splunk for F5
Pete Silva & Lori MacVittie both had blog posts last week featuring the F5 Application for Splunk, so I thought I’d take the opportunity to get Splunk installed and check it out. In this first part, I’ll cover the installation process. This is one of the easiest installions I've ever written about--it's almost like I'm cheating or something. Installing Splunk My platform of choice for this article is Ubuntu, so I downloaded the 4.2.1 Debian package for 64-bit systems from the Splunk site. Installation is a one step breeze: dpkg –i /var/tmp/splunk-4.2.1-98165-linux-2.6-amd64.deb After installation (defaulting to /opt/splunk) start the Splunk server: /opt/splunk/bin/splunk start I had to accept the license agreement during the startup process. Afterwards, I was instructed to point my browser to http:<server>:8000. I logged in with the default credentials (admin / changeme) and then was instructed to change my password, which I did (you can skip this step if you prefer). Pretty easy path to an completed installation. The browser should now be in the state shown below in Figure 1. Installing Splunk for F5 Click on Manager in the upper right-hand corner of the screen, which should take you to the screen shown below in Figure 2. Next, click on Apps as shown below in Figure 3. At this point you have a choice. If you downloaded the Splunk for F5 app from splunkbase, you can click the “install app from file” button. I chose to install from the web, so I clicked the “find more apps online” button. This loaded a listing from splunkbase, with the Splunk for F5 app shown at the bottom of Figure 4 below. After clicking the “install Free” button, I had to enter my splunk.com credentials, then the application installed. Splunk requested a restart, so I restarted and then logged back in. My new session was returned to the online apps screen, so to get to my new F5 app, I clicked “back to search” in the upper left corner, which took my to the Search app home page. Finally, in the upper right corner I selected App and then clicked “Splunk for F5 Security”. This resulted in the screen show below in Figure 5. Success! Now…what to do with it? How is this useful? Check back for part two next week… For some hints, check out the blogs I mentioned at the top of this article from Pete and Lori: Spelunking for Big Data Do You Splunk 2.0 Other Related Articles Do you Splunk? ASM & Splunk integration - DevCentral - F5 DevCentral > Community ... F5 Networks Partner Spotlight - Splunk f5 ltm dashboard in splunk - DevCentral - F5 DevCentral ... Logging HTTP traffic to Splunk - DevCentral - F5 DevCentral ... Client IP Logging with F5 & Splunk - DevCentral - F5 DevCentral ...Splunk Log Publisher with iRule HSL
Hi, I'm having some difficulty understanding the relationship between the HSL iRule commands and formatted log publishers (Splunk, in this case) in 11.5.1. Sorry if the formatting isn't what the community expects - I'm fairly new to this ecosystem. Setup One node called "splunk-universal-forwarder" running a Splunk universal forwarder listening on 9996/tcp. A pool called "remote-logging-pool" with the splunk-universal-forwarder node listening on 9996/tcp. Log Destination "splunk-rhsl" of type Remote HSL pointed at the remote-logging-pool over TCP. Log Destination "splunk-formatted" of type "Splunk" forwarding to "splunk-rhsl" Log Publisher "splunk-publisher" pointing at the splunk-formatted destination. Log Publisher "rhsl-publisher" pointing at the splunk-rhsl destination. A virtual server backed by a pool with a single node serving up content http://10.1.1.2 (the IP address is irrelevant for this question). Given the iRule: when CLIENT_ACCEPTED { set hsl [HSL::open -publisher /tst/splunk-publisher] set hsl [HSL::open -proto TCP -pool remote-logging-pool] set hsl [HSL::open -publisher /tst/Rhsl-publisher] } when HTTP_REQUEST { HSL::send $hsl "<190>|[IP::local_addr]|[HTTP::uri]\n" } The bottom two set statements in the when CLIENT_ACCEPTED command result in a messaging going successfully to the Splunk forwarder. The first set statement results in no messages going to the Splunk forwarder. Questions Are HSL commands in iRules unable to use formatted publishers? What exactly does a formatted publisher do? It seems like it would write data to the remote receiver in a format that particular receiver expects, but I haven't found a lot of documentation on this yet. Is there an interactive way to generate dummy log events to send to different destinations and publishers to see what the effect is on remote receiver? Thanks for your help!Lightboard Lessons: Application Visibility and Reporting
Application Visibility and Reporting (AVR) is a module that lets you analyze performance of web applications. It provides detailed metrics and statistics about application traffic running through the BIG-IP system. AVR allows you to set up an analytics profile that will show you in-depth statistics on various metrics like server latency, page load time, throughput as well as entities on the BIG-IP like Client IP addresses, requested URLs, response codes, user agents, and many more. AVR also has robust notification capabilities that allow internal and external logging, SNMP traps, and email notification. One of our rock star F5 employees, Ken Bocchino, wrote an iApp that utilizes the power of AVR reporting in the Splunk application...really cool stuff! Check out the video below to learn more about AVR and how it can help you gain much-needed visibility over the performance of your web applications. Related Resources: Setting Up Application Statistics Collection F5 Analytics iAppAPM integration with splunk
Hey, i configured the free version of splunk and managed to get the \var\log\apm files, i can see the logs but the built-in dashboard for apm isnt showing data (except one chart) i think its because we run version 13 and this template of splunk was built for 11.6 is anyone tried to modify this template ? or found other creative solution ? i have to make this thing work!External Reporting with BIG-IP ASM
We all know that the BIG-IP ASM does a bunch of great things to protect web applications from malicious attackers. We also know that it’s extremely important to review logs to ensure visibility and awareness about application traffic. In fact, the OWASP organization published the “OWASP Top 10 Proactive Controls” as recommended steps to help mitigate each of their published Top 10 security vulnerabilities. Some of these proactive controls help mitigate more than one of the Top 10 vulnerabilities, but they list “Logging” as a control that helps mitigate all ten! They say it like this, “Logging and tracking security events and metrics helps to enable "attack-driven defense" making sure that your security testing and controls are aligned with real-world attacks against your system.” Needless to say, event logging is critical to the overall security of your web applications. BIG-IP ASM Logging Profiles The BIG-IP ASM dedicates significant resources to event logging. After all, when the ASM blocks a malicious request, you’ll likely want to know all the details associated with that request. Fortunately, you can create a “logging profile” and configure it to capture all kinds of great information. To create a logging profile, navigate to Security >> Event Logs >> Logging Profiles and then click the “Create” button on the upper right part of the page to begin creating a new logging profile. Check out the screenshot below to see the details: You can name your profile whatever you want, but in this case, we are going to set up a profile to send logs to Splunk, so I named mine “Splunk”. Because I’m sending these logs to a remote server, I enabled the “Remote Storage” checkbox and it gave me all the different options for choosing format, protocol, server address/port, and storage format. I chose the UDP protocol because that’s what Splunk uses, I typed in the IP address for the Splunk server I’m using (be sure to click the “Add” button after you input IP address and port), and I moved all the Storage Format items from “Available” to “Selected”. Finally, click the “Finished” button at the bottom of the screen to complete the profile creation. The screenshot below shows all the completed configurations for my profile: Now we have a complete profile that will send all the “Selected” items to the Splunk server using UDP port 514. There’s still one thing to do, though. We need to associate this profile with a virtual server on the BIG-IP. After all, the logging profile won’t know which server to be capturing log information for unless we tell it. To do this, navigate to Local Traffic >> Virtual Servers >> Virtual Server List and click on the virtual server you want to associate with this Logging Profile. When you select the virtual server, you’ll notice a series of menu choices across the top of the screen…click on the “Security” link and select the “Policies” option. When you do that, you’ll see the screen shown below: Notice that I have a security policy (auction_security_policy) and it is enabled on this virtual server. Additionally, I changed the “Log Profile” option from Disabled to Enabled. When it changes to Enabled, you’ll see a menu that allows you to move logging profiles from “Available” to “Selected”. I moved my Splunk logging profile to the Selected column and finally clicked the “Update” button. Now, I have a fully functional logging profile that has been enabled on my virtual server. It’s time to generate some logs by visiting (or, in my case, attacking) the web application that is being protected by my Application Security Policy. I’ll save you all the screenshots of my web application, but I will show you the log report captured on the BIG-IP when I attempted an injection attack against my web application. Navigate to Security >> Event Logs >> Application >> Requests and you’ll see the list of illegal requests that were blocked by the ASM security policy. Remember how the logging profile listed the “Remote Storage” as an option? Well, if you looked closely, you noticed that it also included “Local Storage” as well. That means the BIG-IP will keep a record of all these illegal requests in addition to sending them over to the Remote Storage server. You can obviously configure these settings differently depending on your needs. Anyway, back to the illegal request list. Notice that two of the ASM attack signatures detected a violation in the request. And, rightfully so…after all, I was attempting an injection attack! After I attempted the injection attack, I wanted to slightly change the configuration of the logging profile, so I changed the logging profile to only capture attack-type for each illegal request. I navigated back to my Splunk logging profile and moved everything but “attack_type” back from Selected to Available and then clicked Update. See the screenshot below: The nice thing about making this change in the logging profile is that I don’t have to change it anywhere else on the BIG-IP. Any virtual server that has this logging profile enabled will now only capture the attack_type field when logging an illegal request. So, enough with the BIG-IP…what about the remote Splunk server? Well, I logged into Splunk and set up a custom search on UDP port 514 since that is the port I configured to send logs to Splunk from the BIG-IP. Check out the screenshot below to see the details that Splunk captured: Notice the first request in Splunk lists all the details of the illegal request. If you look closely, you’ll notice that all these details match up to the list of “Selected” items chosen in the logging profile. Then, notice that the most current request (the one at the top of the list) only lists the attack_type…in this case it’s “SQL-Injection”. Pretty cool stuff, huh? The BIG-IP is capable of sending log information to much more than just Splunk, so get out there and configure those logging profiles to suite your specific needs. Remember, log review is one of the most critical things you can do to protect all your web applications!
ASM Logging to Splunk Anomoly
Hello, The ASM logs we're sending to Splunk have random (Splunk assigned?) field names. For example, violation_rating is named cn2 in Splunk, attack_type shows up as cs4, user_agent is called pm_fpua in Splunk, and so on. Does anyone know if this is a Splunk issue or a logging profile issue? The profile I inherited was configured with a logging format of Common Event Format (ArcSight) although we're talking to Splunk. I assumed changing it back to Key-Value Pairs (Splunk) might fix the issue but it's still jacked up. Thanks, ToneF5 Splunk logging issue
Hi Team, We have a requirement where all of the F5s in our environment need to send logs to the Splunk VIP configured on a different F5. I want to use F5 Analytics iApp to simplify the deployment. The Splunk VIP (on a different F5) has 2 Splunk servers as the Pool members. When we are creating "HTTP Event Collector" on Spunk, it generates a Token which I need to use in the iApp. However, the issue is each Splunk server is generating a separate Token (so total 2 tokens) whereas the iApp is accepting only one token. How can we resolve this issue? Thanks in advance!! Regards Vijay
