Getting Started with Splunk for F5
Pete Silva & Lori MacVittie both had blog posts last week featuring the F5 Application for Splunk, so I thought I’d take the opportunity to get Splunk installed and check it out. In this first part, I’ll cover the installation process. This is one of the easiest installions I've ever written about--it's almost like I'm cheating or something. Installing Splunk My platform of choice for this article is Ubuntu, so I downloaded the 4.2.1 Debian package for 64-bit systems from the Splunk site. Installation is a one step breeze: dpkg –i /var/tmp/splunk-4.2.1-98165-linux-2.6-amd64.deb After installation (defaulting to /opt/splunk) start the Splunk server: /opt/splunk/bin/splunk start I had to accept the license agreement during the startup process. Afterwards, I was instructed to point my browser to http:<server>:8000. I logged in with the default credentials (admin / changeme) and then was instructed to change my password, which I did (you can skip this step if you prefer). Pretty easy path to an completed installation. The browser should now be in the state shown below in Figure 1. Installing Splunk for F5 Click on Manager in the upper right-hand corner of the screen, which should take you to the screen shown below in Figure 2. Next, click on Apps as shown below in Figure 3. At this point you have a choice. If you downloaded the Splunk for F5 app from splunkbase, you can click the “install app from file” button. I chose to install from the web, so I clicked the “find more apps online” button. This loaded a listing from splunkbase, with the Splunk for F5 app shown at the bottom of Figure 4 below. After clicking the “install Free” button, I had to enter my splunk.com credentials, then the application installed. Splunk requested a restart, so I restarted and then logged back in. My new session was returned to the online apps screen, so to get to my new F5 app, I clicked “back to search” in the upper left corner, which took my to the Search app home page. Finally, in the upper right corner I selected App and then clicked “Splunk for F5 Security”. This resulted in the screen show below in Figure 5. Success! Now…what to do with it? How is this useful? Check back for part two next week… For some hints, check out the blogs I mentioned at the top of this article from Pete and Lori: Spelunking for Big Data Do You Splunk 2.0 Other Related Articles Do you Splunk? ASM & Splunk integration - DevCentral - F5 DevCentral > Community ... F5 Networks Partner Spotlight - Splunk f5 ltm dashboard in splunk - DevCentral - F5 DevCentral ... Logging HTTP traffic to Splunk - DevCentral - F5 DevCentral ... Client IP Logging with F5 & Splunk - DevCentral - F5 DevCentral ...Lightboard Lessons: Application Visibility and Reporting
Application Visibility and Reporting (AVR) is a module that lets you analyze performance of web applications. It provides detailed metrics and statistics about application traffic running through the BIG-IP system. AVR allows you to set up an analytics profile that will show you in-depth statistics on various metrics like server latency, page load time, throughput as well as entities on the BIG-IP like Client IP addresses, requested URLs, response codes, user agents, and many more. AVR also has robust notification capabilities that allow internal and external logging, SNMP traps, and email notification. One of our rock star F5 employees, Ken Bocchino, wrote an iApp that utilizes the power of AVR reporting in the Splunk application...really cool stuff! Check out the video below to learn more about AVR and how it can help you gain much-needed visibility over the performance of your web applications. Related Resources: Setting Up Application Statistics Collection F5 Analytics iAppExternal Reporting with BIG-IP ASM
We all know that the BIG-IP ASM does a bunch of great things to protect web applications from malicious attackers. We also know that it’s extremely important to review logs to ensure visibility and awareness about application traffic. In fact, the OWASP organization published the “OWASP Top 10 Proactive Controls” as recommended steps to help mitigate each of their published Top 10 security vulnerabilities. Some of these proactive controls help mitigate more than one of the Top 10 vulnerabilities, but they list “Logging” as a control that helps mitigate all ten! They say it like this, “Logging and tracking security events and metrics helps to enable "attack-driven defense" making sure that your security testing and controls are aligned with real-world attacks against your system.” Needless to say, event logging is critical to the overall security of your web applications. BIG-IP ASM Logging Profiles The BIG-IP ASM dedicates significant resources to event logging. After all, when the ASM blocks a malicious request, you’ll likely want to know all the details associated with that request. Fortunately, you can create a “logging profile” and configure it to capture all kinds of great information. To create a logging profile, navigate to Security >> Event Logs >> Logging Profiles and then click the “Create” button on the upper right part of the page to begin creating a new logging profile. Check out the screenshot below to see the details: You can name your profile whatever you want, but in this case, we are going to set up a profile to send logs to Splunk, so I named mine “Splunk”. Because I’m sending these logs to a remote server, I enabled the “Remote Storage” checkbox and it gave me all the different options for choosing format, protocol, server address/port, and storage format. I chose the UDP protocol because that’s what Splunk uses, I typed in the IP address for the Splunk server I’m using (be sure to click the “Add” button after you input IP address and port), and I moved all the Storage Format items from “Available” to “Selected”. Finally, click the “Finished” button at the bottom of the screen to complete the profile creation. The screenshot below shows all the completed configurations for my profile: Now we have a complete profile that will send all the “Selected” items to the Splunk server using UDP port 514. There’s still one thing to do, though. We need to associate this profile with a virtual server on the BIG-IP. After all, the logging profile won’t know which server to be capturing log information for unless we tell it. To do this, navigate to Local Traffic >> Virtual Servers >> Virtual Server List and click on the virtual server you want to associate with this Logging Profile. When you select the virtual server, you’ll notice a series of menu choices across the top of the screen…click on the “Security” link and select the “Policies” option. When you do that, you’ll see the screen shown below: Notice that I have a security policy (auction_security_policy) and it is enabled on this virtual server. Additionally, I changed the “Log Profile” option from Disabled to Enabled. When it changes to Enabled, you’ll see a menu that allows you to move logging profiles from “Available” to “Selected”. I moved my Splunk logging profile to the Selected column and finally clicked the “Update” button. Now, I have a fully functional logging profile that has been enabled on my virtual server. It’s time to generate some logs by visiting (or, in my case, attacking) the web application that is being protected by my Application Security Policy. I’ll save you all the screenshots of my web application, but I will show you the log report captured on the BIG-IP when I attempted an injection attack against my web application. Navigate to Security >> Event Logs >> Application >> Requests and you’ll see the list of illegal requests that were blocked by the ASM security policy. Remember how the logging profile listed the “Remote Storage” as an option? Well, if you looked closely, you noticed that it also included “Local Storage” as well. That means the BIG-IP will keep a record of all these illegal requests in addition to sending them over to the Remote Storage server. You can obviously configure these settings differently depending on your needs. Anyway, back to the illegal request list. Notice that two of the ASM attack signatures detected a violation in the request. And, rightfully so…after all, I was attempting an injection attack! After I attempted the injection attack, I wanted to slightly change the configuration of the logging profile, so I changed the logging profile to only capture attack-type for each illegal request. I navigated back to my Splunk logging profile and moved everything but “attack_type” back from Selected to Available and then clicked Update. See the screenshot below: The nice thing about making this change in the logging profile is that I don’t have to change it anywhere else on the BIG-IP. Any virtual server that has this logging profile enabled will now only capture the attack_type field when logging an illegal request. So, enough with the BIG-IP…what about the remote Splunk server? Well, I logged into Splunk and set up a custom search on UDP port 514 since that is the port I configured to send logs to Splunk from the BIG-IP. Check out the screenshot below to see the details that Splunk captured: Notice the first request in Splunk lists all the details of the illegal request. If you look closely, you’ll notice that all these details match up to the list of “Selected” items chosen in the logging profile. Then, notice that the most current request (the one at the top of the list) only lists the attack_type…in this case it’s “SQL-Injection”. Pretty cool stuff, huh? The BIG-IP is capable of sending log information to much more than just Splunk, so get out there and configure those logging profiles to suite your specific needs. Remember, log review is one of the most critical things you can do to protect all your web applications!Do You Splunk 2.0
A little over two years ago I blogged Do you Splunk? about the reporting integration with our FirePass SSL VPN and BIG-IP ASM. The Splunk reports have provided customers valuable insight into application access and user behavior along with deep analysis of application violations, web attacks and other key metrics. Recently, Splunk and F5 have been working behind the scenes and now you can also get 22 different templates for detailed reporting on the BIG-IP Access Policy Manager. BIG-IP APM is a flexible, high-performance access and security solution that runs as a module on BIG-IP LTM. Splunk is the data engine for IT. It collects, indexes and harnesses the fast-moving IT data generated by all of your IT systems and infrastructure - whether physical, virtual or in the cloud and correlates various pieces of data sources to provide new views and new insights. Splunk makes it possible to search and navigate data from any application, server or network device from a web browser, in real time. Logs, configurations, messages, traps, alerts, and scripts: if a machine generates it, Splunk will index it. The Splunk for F5 App provides real-time dashboards for monitoring key performance metrics. Reports from Splunk support long-term trending and can be downloaded in PDF or Excel formats or scheduled for email delivery. The F5 App supports core Splunk functionality such as deep drill-down from graphical elements, robust role-based access controls and Splunk’s award-winning search capabilities. The following are a sample of the reports available in this version of Splunk for F5 using ASM, APM and FirePass data: Request Status Over Time Top Attacker Top Sites Top Violations Active Sync by Device Type Top Device Type Top User Geo-location Reports Session Duration and Throughput Authentication Success/Failure Connections by User Failed Connections by User All Connections Over Time Splunk also has the unique ability to augment data from FirePass and ASM by connecting to and gathering data from Active Directory or LDAP and asset management databases that can highlight asset or application owner information. Businesses are faced with competing challenges when it comes to granting their mobile workforce access to company data. The data must be readily accessible to users on the go but at the same time companies must protect and safeguard their internal systems that contain sensitive information. Robust monitoring controls are a must for maintaining auditing access, enabling dynamic application access and preventing data loss and availability issues. Resources: Splunk for F5 F5 Networks Partner Spotlight - Splunk Knowledgebase: Splunk for Use with F5 Networks Solutions Video: Splunk for Use with F5 Networks Solutions Splunk Templates for BIG-IP Access Policy Manager (pdf) Splunk for FirePass SSL VPN (pdf) Splunk for Application Security Manager (pdf) ASM & Splunk integration F5 Security Community Group on DevCentral Do you Splunk?A Catch from the Codeshare: F5 Analytics iApp
On the side of the road in northern Missouri just north of Mark Twain’s stomping grounds, there is aslice of hillside removed just to the side of the highway. In Arkansas, there’s anondescript field tucked away in a state park. Short of word of mouth and this thing they call the internet, you wouldn’t be any the wiser that buried or surfaced in these two locations are an amazing variety of geodes and diamonds, respectively. In this article series I will explore recent and well-aged gems from the codeshare, highlighting inventive solutions contributed by you, the community. Please join me on this great adventure as we oscillate on the Mohs’ scale of codeshare geekery. I’ve been careful to focus most entries on community contributed content, but this release 10 days ago is too cool for school, so I’ll make an exception for this F5 contributed Analytics iApp from Ken Bocchino. It is early access with some open items still under development, but this iApp is the outcome of a close partnership with F5 and Splunk to take analytics data from BIG-IP and display the various data points within the Splunk F5 Application. A couple screen shots from the app: This iApp has been tested on BIG-IP versions 11.4 through 12.0. Take a peek at the installation video and then download the iApp! Thanks for checking out this latest catch from the codeshare!
F5 Friday: Performance Analytics–More Than Eye-Candy Reports
#v11 Application-centric analytics provide better visibility into performance, capacity and infrastructure utilization Maintaining performance and capacity of web sites and critical applications – especially those of the revenue-generating ilk – can be particularly difficult in complex environments. The mix of infrastructure and integration can pose problems when trying to determine exactly where capacity may be constrained or from where performance troubles are originating. Visibility into the application delivery chain is critical if we are to determine where and at what points in the chain performance is being impaired or constraints on capacity imposed, perhaps artificially. The trouble is that the end-user view of application performance tells you only that a site is or isn’t not performing up to expectations. It’s myopic in that sense, and provides little to no assistance in determining why an application may be performing poorly. Is it the Internet? An external component? An integration point? A piece of the infrastructure? Is it because of capacity (load on servers) or a lack thereof? It’s easy to point out a site is performing poorly or has unacceptable downtime, it’s quite another to determine why such that operations and developers can resolve the problem. Compounding the difficulties inherent in application performance monitoring is the way in which network infrastructure – including application delivery controllers – traditionally report statistics related to performance. It’s very, well, network-focused – with reports that provide details on network-oriented objects such as Virtual IP addresses and network segments. This is problematic because when a specific application is having issues, you want to drill down into the application, not a shared IP address or network segment. And you wouldn’t be digging into performance data if you didn’t already know there was a problem. So providing a simple “total response time” report isn’t very helpful at all in pinpointing the root cause. It’s important to be able to break out, by application, a more granular view of performance that provides insight into client, network, and server-side performance. iApp Analytics BIG-IP v11 introduced iApp, which addresses a real need to provision and manage application delivery services with an application-centric perspective. Along with iApp comes iApp Analytics, a more application-centric view of performance and capacity-related data. This data provides a more holistic view of performance across network, client and server-side components that allows operations to drill down into a specific application and dig around in the data to determine from where performance problems may be originating. This includes per URI reporting, which is critical for understanding API usage and impact on overall capacity. For organizations considered more advanced architectural solutions to addressing performance and capacity, this information is critical. For example, architecting scalability domains as part of a partitioning or hyper-local scalability pattern will need to have a detailed understanding of per-URI utilization. Being able to tie metrics back to a specific business application enables IT to provide a more accurate operational cost to business stakeholders, which enables more a accurate ROI analysis and proactive approach to addressing potential growth issues before they negatively impact performance or worse, availability. Transactions can be captured for diagnosing customer or application issues. Filters make it easy to narrow down the results to specific user agents, geographies, pools or even a client IP. The ability to view the headers and and the response data can shave valuable time off identifying problems. Thresholds on a per application, virtual server or pool member can be configured to identify if transactions or throughput levels increase or decrease significantly. This can be an early warning sign that problems are occurring. An alert can be delivered via syslog, SNMP or email when these thresholds are exceeded. Viewing of analytics can be accomplished through iApp application-specific view which provides the context of the associated business application. Metrics can also be delivered to an off-box SIEM solution such as Splunk. While detailed, per-application performance and usage data does, in fact, make for very nice eye-candy reports, such data is critical to reducing the time associated with troubleshooting and enabling more advanced, integrated scalability-focused architectures to be designed and deployed. Because without the right data, it’s hard to make the right decisions for your applications and your infrastructure. Happy Performance Monitoring! Infrastructure Scalability Pattern: Partition by Function or Type Lots of Little Virtual Web Applications Scale Out Better than Scaling Up Forget Hyper-Scale. Think Hyper-Local Scale. F5 Friday: You Will Appsolutely Love v11 Introducing v11: The Next Generation of Infrastructure BIG-IP v11 Information Page F5 Monday? The Evolution To IT as a Service Continues … in the Network F5 Friday: The Gap That become a Chasm All F5 Friday Posts on DevCentral ABLE Infrastructure: The Next Generation – Introducing v11F5 Friday: Spelunking for Big Data
Managing the other kind of performance in a data center requires the ability to analyze a whole lotta data. Big operational data. “Big data” right now is nearly as hyped as cloud computing . The vast amounts of data collected that need to be shared, integrated, replicated, backed up, and managed is growing at a phenomenal rate. But when folks talk about “big data” they’re focused primarily on application data, on user-generated data, on business data. They are not generally concerned with the other “big data” that threatens to overwhelm data center operations on a daily basis: operational data. Every day, in data centers across the world, gigabyte upon gigabyte of log data is generated. Some of it is mundane bandwidth and throughput data. Some of it is routine web application data, reporting on number of requests received in any given period of time. Other data contains more gnarly information, such as who and what device was trying to inject malicious code into a web application. It’s all important data, and when you combine the gigabytes of log files from just about every device in the data center, well, that’s BIG data. Without the means to aggregate, search, and analyze all that data as a view of “the data center” (as opposed to individual components), however, it’s just bits and bytes and wasted disk. Administrators and operators need a way to aggregate and correlate events across the entire data center so they can more easily find and understand any given event or problem that may be occurring as well as providing a holistic view of data center performance. And by performance I mean not just “how fast does my application go” but “how well is my web application firewall performing its responsibilities.” After all, one of the ways in which IT justifies the acquisition of solutions is by providing a Return On Investment (ROI) based on the solution performing its intended task. MANAGING the OTHER KIND of PERFORMANCE If you deploy F5 BIG-IP Access Policy Manager (APM) as an access management solution, you’d like to know that it’s actually doing just that – and how well it’s doing it. Without that data it’s hard to compute the ROI and provide the business with “proof” that its investments in data center solutions are paying back the organization as expected. The problem is that while individual solutions may report on how well they are performing, they are unlikely to integrate and correlate data from other systems to provide a holistic view of “the other kind of performance.” That’s where those standards and management solutions come in handy. Leveraging standards and integration methods to aggregate data from across data center components and even data centers (including cloud computing providers), solutions exist that can provide the visibility into the “other kind” of performance of data center components necessary to understand not only how each component is performing but also see the “big picture” across the entire data center. Now, the way in which you paint that big picture differs. You can, of course, go spelunking through the data center yourself to find the data you need and manually aggregate it. Such manual processes do not scale well, of course, and as data grows so does the time and effort required to perform such a task. The big operational data in today’s data centers makes that a Herculean task that, on reflection, you’ll find is probably much better suited to an automated solution. A good option is a solution like Splunk, which phonetically sounds a whole lot like “spelunk” and unsurprisingly that’s not just coincidence. What Splunk does is exactly what you may think it does: it explores the entire data center, indexing and aggregating and correlating data from just about every kind of system, platform, and device. Not only does it provide a single-point of entry into the “big data” of enterprise infrastructure, but it also allows analysis of that data from simple to complex queries, enabling operators and admins to fully explore the depths of big data in the enterprise from the comfort of their console. Now available (for free, as in gratis) is Splunk for F5 (Version 2.0). Not only does this version support APM, but also includes integrated data from F5 BIG-IP Application Security Manager (ASM) and FirePass as well. For more details on this offering, please check out fellow blogger Pete Silva’s latest post, “Do You Splunk 2.0”. Happy Spelunking! Splunk for F5 Do You Splunk 2.0 F5 Friday: Protocols are from Venus. Data is from Mars. All F5 Friday Entries on DevCentral Video: Splunk for Use with F5 Networks Solutions Splunk Templates for BIG-IP Access Policy Manager Splunk for FirePass SSL VPN Splunk for Application Security Manager ASM & Splunk integration F5 Security Community Group on DevCentral
