cancel
Showing results for 
Search instead for 
Did you mean: 
Login & Join the DevCentral Connects Group to watch the Recorded LiveStream (May 12) on Basic iControl Security - show notes included.
Greg_Coward
F5 Employee
F5 Employee

The new Splunk Add-on for F5 BIG-IP includes several objects, (modular inputs, CIM-knowledge, etc.) that work to “normalize” incoming BIG-IP data for use with other Splunk apps, such as Splunk Enterprise Security and the Splunk App for PCI Compliance.   

The add-on includes a mechanism for pulling network traffic data, system logs, system settings, performance metrics, and traffic statistics from the F5 BIG-IP platform using F5’s iControl API, (see below).

0151T000003q8IqQAI.png

But what I'm really excited about is that the add-on now integrates with F5 Telemetry Streaming, (TS). With TS I am easily able to declaratively aggregate, normalize, and push BIG-IP statistics and events, (JSON-formatted) to a variety of third-party analytics vendors. 

For the remainder of this article, we’ll take a look at how I integrate F5 TS with Splunk Enterprise. I’ll be working with an existing BIG-IP deployment as well as a newly deployed Splunk Enterprise instance. As an added bonus, (and since it’s part of the article’s title) I’ll import a couple custom dashboards, (see below) to visualize our newly ingested telemetry data.

Oh! As an "Extra" added bonus, here is a link to a video walk through of this solution.  

0151T000003q8IvQAI.png

Installing the Splunk Add-on for F5 BIG-IP and Splunk CIM

Installing the Splunk F5 add-on is very simple. Additionally, to make use of the add-on I’ll need to install Splunk’s Common Information Model, (CIM).   

1.    From the top Splunk the search page, I select ‘Apps’ → ‘Find More Apps’.  0151T000003q8J0QAI.png

2.   I browse for “CIM” and select the Splunk Common Information Model add-on.

3.   I accept the license agreement, provide my Splunk account login credentials and select ‘Login and Install’.

0151T000003q8IwQAI.png

4.   I’ll repeat steps 2-3 to install the Splunk Add-on for F5 BIG-IP. 

0151T000003q8J1QAI.png

Setup Splunk HTTP Event Collector

To receive incoming telemetry data into my Splunk Enterprise environment over HTTP/HTTPs I will need to create an HTTP Event Collector.

1.    From the UI I select ‘Settings’ → ‘Data Inputs’. I select ‘HTTP Event Collector’ from the input list.

2.   Prior to creating a new event collector token, I must first enable token access for my Splunk environment. On the ‘HTTP Event Collector’ page, I select ‘Global Settings’. I set ‘All Tokens’ to enabled, default index, incoming port and ensure SSL is enabled. I click ‘Save’ to exit.

0151T000003q8IxQAI.png

3.    I select ‘New Token’ and provide a name for the new collector and select ‘Next’.

0151T000003q8J5QAI.png

4.    On the ‘Input Settings’ tab I’ll select my allowed index(es) and select ‘Review’ then ‘Submit’.

5.    Once the token is created, I will need to copy the token for use with my F5 TS configuration.

0151T000003q8J6QAI.png

Configure Telemetry Streaming

With my Splunk environment ready to receive telemetry data, I now turn my attention to configuring the BIG-IP for telemetry streaming. Fortunately, F5’s Automation Toolchain configuring the BIG-IP is quite simple.  

1.    I’ll use Postman to POST an AS3 declaration to configure telemetry resources, (telemetry listener, log publisher, logging profiles, etc.).  

0151T000003q8JAQAY.png

The above AS3 declaration, (available here) deploys the required BIG-IP objects for pushing event data to a third-party vendor. Notably, it creates four (4) logging profiles I’ll attach to my application’s virtual server.

2.    Still using Postman, I POST my TS declaration, (sample). I will need to provide my Splunk HTTP Collector endpoint address/port as well as the token generated previously.

0151T000003q8MJQAY.png

Associate Logging Profiles to Virtual Server

The final step to configuring the BIG-IP for telemetry streaming is associating the logging profiles I just created with my existing virtual server. In addition to system telemetry, these logging profiles, when assigned to a virtual, will send LTM, AVR, and ASM telemetry.

1.    From the BIG-IP management UI, I select ‘Local Traffic’ → ‘Virtual Servers’ → <virtual>.

0151T000003q8J7QAI.png

2.    Under ‘Configuration’ I select ‘Advanced’, scroll down and select the HTTP, TCP, and request logging profiles previously created. I select ‘Update’ at the bottom of the page to save

0151T000003q8J8QAI.png

3.   From the top of the virtual server page, I select ‘Security’ → ‘Policies’. From the policy settings page, I can see that there is an existing WAF policy associated with my application. To enable ASM logging, I select the previously created ASM logging profile from the available logging profiles and select ‘Update’ to save my changes.

0151T000003q8J9QAI.png

With the configuration process complete, I should now start seeing event data in my Splunk Environment.  

Import Dashboards

“Ok, so I have event data streaming into my Splunk environment; now what?” 

Since I have installed the Splunk F5 add-on, I can integrate my “normalized” data with other data sources to populate various Splunk applications like Splunk Enterprise Security and Splunk App for PCI Compliance. Likewise, I can use dashboards to visualize my telemetry data as well as monitor BIG-IP resources/processes. To finish up, I’ll use the following steps to create custom dashboards visualizing BIG-IP metrics and Advanced WAF, (formerly ASM) attack information.

1.    From the Splunk Search page, I navigate to the Dashboards page by selecting ‘Dashboards’.

2.   Select ‘Create New Dashboard’ from the Dashboards page.

0151T000003q8JBQAY.png

3.   Provide a name for the new dashboard and select ‘Create Dashboard’. The dashboard name, (ID will remain unchanged) will be updated in the next step where I replace the newly created dashboard’s XML source with one of the community-supported dashboard XML files here.

4.   On the ‘Edit Dashboard' screen I select ‘Source’ to edit the dashboard XML. I replace the existing XML data with the contents of the ‘advWafInsights.xml’ file. Select ‘Save’ to install the new dashboard.  

0151T000003q8JKQAY.png

 

0151T000003q8J2QAI.png

5.    I’ll repeat steps 1-4 using ‘bigipSystemMetrics.xml’ to install the BIG-IP metrics dashboard,

0151T000003q8J3QAI.png

 

Additional Links

·     F5 Telemetry Streaming

·     Splunk Add-on for F5 BIG-IP

·     Splunk Common Information Model 

·     F5 Automation Toolchain

Comments
Dojs
Cirrostratus
Cirrostratus

WOW amazing doc man. Congratulations

asamadyar
Nimbostratus
Nimbostratus

Thanks for your helpful post buddy. That was perfect.

I have a problem with Postman. When I post, I get "404 Not Found" and just "1" shown in response body. When I tried to open the url (https://172.16.1.1/mgmt/shared/appsvcs/declare), this message is displayed: "{"code":404,"message":"Public URI path not registered: /shared/appsvcs/declare","referer":"172.16.10.10","restOperationId":.............,"kind":":resterrorresponse"}"

The same thing happens with the second url. (https://172.16.1.1/mgmt/shared/telemetry/declare?show=detail)

 

Would you please help me?

Thanks in advance.

Ali

aheilmaier
Nimbostratus
Nimbostratus

 I have not tried AS3 streaming declaration (..appsvcs/declare...), only the telemetry declarations.

I suggest you to have a look on telemetry documentation https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/quick-start.html

aheilmaier
Nimbostratus
Nimbostratus

Thank's  super interesting article.

i tried ts with the telemetry declaration only.

What format type have you used ?

 

I used legacy, because multi-metric was not available in the older documentation.

And legacy was simpler to parse in Splunk search commands and also with splunk streaming stats.

Umesh_Shetty
Altostratus
Altostratus

I wanted to understand what happens behind the scenes when a Virtual server with an IP address of 255.255.255.254 is configured ? How are the logs forwarded to this virtual server and how does this VS integrate with Telemetry streaming to forward these logs to Splunk ?

Coreyf311
Nimbostratus
Nimbostratus

@asamadyar, I just went through this today. You need to deploy the f5-appsvcs rpm to make the API available.

 

I have this configured except for the "telemetry_asm_security_log_profile" as we do not have ASM licensed. I am not seeing the AVR source in my splunk logs. Can you tell me what might be missing? Is it the ASM log profile?

jomedusa
Altostratus
Altostratus

Have the samples of the declarations been moved, the links no longer work.

Greg_Coward
F5 Employee
F5 Employee

Hi there, thank you for the feedback. I updated the link to reflect the file renaming.

jomedusa
Altostratus
Altostratus

Awesome thank you so much for the excellent article and information...and for your quick response.

jomedusa
Altostratus
Altostratus

We are not getting the AVR source into Splunk either, @Coreyf311, did you figure your issue out, I also don't have the ASM profile configured as we don't have license for this module.

I also noticed that everything was created within the Common/Shared partition/path...I wasn't able to select this as remote publisher as only those that are listed in the Common are availalble.

 

Thanks,

 

Joe

Greg_Coward
F5 Employee
F5 Employee

Hello Joe,

 

There is a TMSH command required to enable AVR. Run this on your BIG-IP(s) - tmsh modify analytics global-settings { external-logging-publisher /Common/Shared/telemetry_publisher offbox-protocol hsl use-offbox enabled }.

 

You can modify the AS3 declaration to create resources in a different partition if necessary. With that said, the resources are deployed as 'Shared' under the Common partition. You should be able to view and attach them to your virtual(s). Note: If you do use a different partition, you will need to adjust the above TMSH command accordingly.

jomedusa
Altostratus
Altostratus

We are still not getting AVR information sent to Splunk but we have created specific consumer for stats items we want to display in Splunk. We disabled the system default one as it would send to much traffic to Splunk for what we are wanting to accomplish. One thing I have noticed is that all nodes within the Device Cluster send data at all times to Splunk for the consumers we have created. Is there a way to tell the "Standby" F5 not to send data as it irrelevant to the actual traffic. With both sets of data coming in we have to know which F5 is active to produce our queries/dashboards.

 

Thanks,

 

Joe

Coreyf311
Nimbostratus
Nimbostratus
I was having intermittent issues with receiving AVR information. I actually left the org that I was doing all this work with. We would see it sometimes and then sometimes it wouldnt produce events. I never got a chance to dive into the issue. We did find the TMSH command Greg referenced to fix the issue and we started getting SOME events but it was "glitchy". Sorry I don't have more to offer.
cornemrc
Nimbostratus
Nimbostratus

Thank you for the article, it helped so much! Since I have activated AVR via tmsh most things are working fine here. Only the ASM events are not available in your dashboard template although the ASM log profile is in place and there are security events produced in our lab.

 

I have inspected the search and found source="f5:bigip:asm" but according to the transforms.conf of the Splunk Addon the source rewrite to this asm source will only match on REGEX = "telemetryEventCategory":"ASM". But ASM raw events have "telemetryEventCategory":"AVR" as you can see here.

 

Have you changed anything in your configuration or is there something I have missed?

Version history
Last update:
‎25-Jan-2021 12:14
Updated by:
Contributors