F5 Analytics iApp

[Ken_Bocchino_49]

Problem this snippet solves:

Analytics iApp v3.7.0

You can use this fully supported version of the analytics iApp template to marshal statistical and logging data from the BIG-IP system. The iApp takes this data and formats it as a JSON object which is then exported for consumption by data consumers, such as F5 BIG-IQ or applications such as Splunk.

The Analytics iApp allows you to configure several categories of data to be exported. For data consumers like Splunk, the iApp lets you configure the network endpoint to which the data is sent.

Version 3.7.0 of the iApp template is fully supported by F5 and available on downloads.f5.com. We recommend all users upgrade to this version. For more information, see https://support.f5.com/csp/article/K07859431.

While this version of the iApp is nearly identical to the v3.6.13 which was available on this page, the major difference (other than being fully supported) is that ability to gather APM statistics using the iApp has been removed from BIG-IP versions prior to 12.0.

Supported/Tested BIG-IP versions: 11.4.0 - 12.1.2.

Data Sources: LTM, GTM, AFM, ASM, APM, SWG, and iHealth (APM statistics require 12.0 or later)

Data Output Formats: Splunk, F5 Analytics, F5 Risk Engine

Splunk App: https://apps.splunk.com/apps/id/f5

The new deployment guide can be found on F5.com: http://f5.com/pdf/deployment-guides/f5-analytics-dg.pdf

• Video Demo - https://player.vimeo.com/video/156773835
  • Solution Architecture - 20s
  • Installation - 1m53s
• UI Demo
  • Device Dashboard - 6m44s
  • Application Issue Troubleshooting - 9m26s
  • Application Team Self Service - 12m17s

Code :

https://downloads.f5.com/esd/ecc.sv?sw=BIG-IP&pro=iApp_Templates&ver=iApps&container=iApp-Templates

[yamashin55]

Thank you Analytics iApp. I tried iApp. can not see Analytics. [Error in 'TsidxStats': WHERE clause is not an exact query] Do you know a cause?

[Ken_Bocchino_49]

@Shingo Yamada can you provide more details on the dashboard and panel displaying this message? I have found that if you're looking at the dashboards immediately after installation of the app you may see some delays and or errors b/c the data models are just starting to be built. You can see the status of the data models under Settings->Data Models and by clicking on the left > to see build status. Also verify you are receiving data into Splunk "index=* source=bigip.* | stats count by index sourcetype source" If you're still having issues feel free to reach out to me via email at KB@F5.com

[yamashin55]

Thank you for your help!!. 「index=* source=bigip.*」 Success !!

[richard_polyak]

Has anybody run into issues running this?

[Tushar_K]

Can you give me the analytics template file

[jspiglerj2rsolves]

Tried deploying the iAPP and received the following error

script did not successfully complete: (script did not successfully complete: ("global-settings" unexpected argument while executing "tmsh::modify [string range $args 7 end] " ("modify" arm line 1) invoked from within "switch -exact -- [string range $args 0 5] { create { tmsh::create [string range $args 7 end] } modify { tmsh::modify [string r..." (procedure "iapp::conf" line 14) invoked from within "iapp::conf modify analytics global-settings avrd-interval 300" invoked from within "if {$::basic__format != "F5 Risk Engine" && $::basic__format != "F5 BIG-IQ" && $::basic__logging == "Yes"} { set deviceinfo "non exist" catch {se..." line:5073) while executing "exec /usr/bin/tmsh -c $command" (procedure "tmsh_exe" line 4) invoked from within "tmsh_exe "create sys application service /Common/${::app}-local { template traffic-group traffic-group-local-only variables replace-all-..." invoked from within "if { $createiapp == "Yes" } {

set existing_iapp "non exist" catch {set existing_iapp [tmsh::get_config sys application service /Common/${::app}-..." line:5052)

Anybody else running into this?

[Philip_Lim_1751]

Nice screenshots. I see the latest version on the Splunk apps page is Version: 0.9.11. Anyone know if this can be done in Splunk Light or download?

[richard_polyak]

Hi Ken,

Question will this iApp work on a GTM provisioned device only? Also could you provide me the latest version of the iApp.

Thx -Rich

[Ken_Bocchino_49]

@Richard, yes this will with a GTM only device.

[Raul_Camacho_30]

I think that may be the problem. We are on 11.1. Will be upgrading to 12.1 very soon. I will try this install again at that time.

[Ken_Bocchino_49]

Yep, that would be the case, the iApp works with versions 11.4.0 and higher.

[adamp_1459]

great iapp however it spams the \var\log\ltm with debug logs,(stats ....) what's the best way to disable the debug notice?

[Ken_Bocchino_49]

@adamp this can be disabled: Do you want to display advanced options? "Yes"; Information Sources -> Log Stats Responses "No"

[adamp_1459]

great thanks alot

[Neil_David_Harr]

Thanks for the quick response, I will wait for a full 24hrs. Will APM session information be available soon?

[Ken_Bocchino_49]

If you have APM sessions on the device you should be seeing that data now, index=* source=bigip.sessiondb

[Neil_David_Harr]

I do not have that source, is this a configuration problem? As mentioned above I used .

I configured the Push SessionDB stats (APM) to yes

[Ken_Bocchino_49]

You have it configured correctly, will verify 12.1.1 APM session status in our lab.

[richard_polyak]

Keith,

Great work on this iApp / Splunk app. I am testing this on about 10 pairs. about half I in splunk the are all the Virtual Servers are reporting up as a health of 0.00. What I am seeing in the F5 logs is the below response

debug scriptd[22114]: 01420004:7: Stats Response for vs_analytics 1484313060 1 400 debug scriptd[22114]: 01420004:7: Stats Response for vs_analytics 1484313120 0 400 debug scriptd[22114]: 01420004:7: Stats Response for vs_analytics 1484313120 1 400

What should I be looking for to resolve this and return a 200?

Thx Rich

[Ken_Bocchino_49]

do you have any ' [ ] etc in virtual descriptions? also try turning off search inside irules within the application mapping section.

[richard_polyak]

Keith so I did some testing today, and luckily I have a lightly used LB pair to work with.

This LB has only 8 Virtual Servers with no special charters in the names or anything in the descriptions. Neither on the pools or nodes. Nodes are named via the IP. We are running 11.5.4 HF2.

If I disable push configuration map then I receive a 200.

This is the format for my Virtual Servers vs_fqdn_port, as an example vs_www.

I went through all my profiles and I do not see anything out of the norm.

Thx Rich

[Ken_Bocchino_49]

Have you attempted to set search iRules = No under the Application Mapping Section?

What does your app mapping section look like, can you send me your mapping export string?

[richard_polyak]

Yes I did try that with no luck.

Below is my mapping

ltm data-group internal vs_analytics-send_stats { app-service /Common/vs_analytics.app/vs_analytics records { application_mapping { data "{10000000000} {App Name~virtual_name~(.*)~Map~~} " } avr_commands {

or mapping export string: ezEwMDAwMDAwMDAwfSB7QXBwIE5hbWV+dmlydHVhbF9uYW1lfiguKil+TWFwfn59IAo=

And I tried removing the (.*) as well.

[Ken_Bocchino_49]

@richard, in working in PM, looks like you needed to add the correct indexes when using the RBAC options. The splunk server was rejecting some of the tenant mapped index names.

[Stephen_Mathez1]

I am seeing the following message repeated in /var/log/ltm:

debug scriptd[32475]: 01420004:7: Stats Response for analytics 1486699800 1 fail

(sometimes it is "0 fail", sometimes "1 fail")

Also, /tmp is filling up with sesslist-* files and I am not seeing anything other than vanilla syslog on the Splunk side. Any suggestions for where to start troubleshooting?

Running 11.5.3 HF2 with APM and using

thanks

[VolvoT_308416]

Hi,

We're also seeing similar logs in the /var/log/ltm. What could be the reason for failure ?

Thanks

[Ken_Bocchino_49]

There are several reasons you could be receiving the "fail" response. this message occurs when the stats send process is unable to get a clean response from the Splunk HEC endpoint. It could be as simple as a connectivity issue to the Splunk server, check to see if you can curl to the server curl -k https://. Verify your protocol type HTTP vs HTTPS. If that is good ensure that the indexes you are using align, i.e. if you're using RBAC a missing index could be the cause. You can also get more details viewing /shared/tmp/"iappname"-stats_output_0 to view the response from the Splunk server.

[VolvoT_308416]

Thanks for the reply. It was simple firewall issue.. F5 was unable to make a connection with the Splunk on 8088 port. Issue resolved...

[whootang]

Does anyone know if you can get this and the F5 App working on the free Splunk trial? i am trying to demo this to management before they sink the big coin for the cloud splunk instance.

Cheers R

[Walter_Kacynski]

Yes, I had it running on an Eval copy of Splunk.

[Shayza_312029]

Hello,

I installed on my i5600, all configuration looks OK. I didn't find any error and in tcpdump I can see that the relevant syslog packets are sending.

The main problem is that I cannot see any relevant information about the i5600 (nothing).

I tried to work with asterisks on the regex, I thought that I may see something, but still, everything is blank. I may concern that it because that I'm working with partitions and the iApp was installed on Common.

someone had the change to make F5/Splunk integration with different partitions ?

Thanks, S

[Ken_Bocchino_49]

Multiple partitions works without issue, the iApp is installed into common. Are you getting 200 OK status from the stats response? Are you seeing any device info in the device dashboard? can you do a index=* | stats count by host source sourcetype index?

[Shayza_312029]

Hi, I'm getting the following event, Log Level:notice Service: scriptd[20602] Status Code: 01420004 Event: Stats Response for SPLUNK 1488265770 0 400

I cannot see any device info in the dashboard.

Regarding to index=* | stats count by host source sourcetype index, I executed it. Seems that there is nothing. I do have regular syslog data in a different service (514), when for the dashboard I'm working with 8808.

[Ken_Bocchino_49]

Sounds like an auth issue when sending the data to Splunk, make sure you have setup HEC correctly. Verify the auth token etc.

[The-messenger]

Ken, have you considered this iapp for VCMP host reporting?

[Ken_Bocchino_49]

yes this also captures VCMP data, you need to install the iapp on the host system and on the guest systems, you will see guest details within the device cluster drilldown,

[The-messenger]

Thanks!

[whootang]

am i having the same problem above running on v13, i followed the video tut and the pdf, but i assume im missing some fundamental setting but cant find it.

showing stats response from splunk 142340** 0 400 showing stats response from splunk 142340** 1 400

[Shayza_312029]

Ryzilla, I followed Ken recommendations. make sure the your HEC setup it right. In my case I was needed changed the auth token.

As I mentioned in my last post, now I have another issue.

Regards,

[whootang]

yeah thanks Shayza, which ones are you referring to? yeah my HFC is setup changed the token a few times to make sure and still no luck

[The-messenger]

Great iapp!

I removed an older version and configured the latest version. In the ltm logs I now see State response fail messages followed by several /Common/ir-splunk_analytics-hec-forwarder-udp-snmptrap - can't read "msg": no such variable while executing "string trimright $msg ",""

[Stephen_Mathez1]

So, I was having connectivity issues which have now been resolved, but I am seeing the following error every 5 minutes. The file names rotate between _0, _1 and _2. The thing is, the files are there and world readable. Any idea what could be causing this?

Script (/Common/splunk.analytics-send_stats) generated this Tcl error: (script did not successfully complete: (could not read "/shared/tmp/splunk.analytics-stats_1": no such file or directory while executing "file size "$filename$currentfile"" ("foreach" body line 24) invoked from within "foreach virtual $virtual_list { set virtual_name "/[tmsh::get_name $virtual]" assign tenant, application, and tier

[mkolozs_236219]

Great APP! I installed v3.6.13 and Splunk app 1.0.0. Unfortunately, I only see partial data for Device Status dashboard. Missing fields are version, build, serial, platform. Any suggestion how to fix this? Other data are there in index=f5-default source =bigip.tmsh.system_status sourcetype = f5:bigip:status:iapp:json

Appreciate in advance.

[jspiglerj2rsolves]

Great app! Alot of potential for being the best ADC visibility app out there on splunk.

One thing I'm having issues with and I think its how the search was constructed is the Application Drill down dashboard, SSL Certificates panel. I can only return the latest certificate object, ssl profile that has been reported to splunk. The search is as follows

| tstats latest(all.cert_name), latest(all.cert_expiration_date), latest(all.cert_expiration_date_human),latest(all.CN) from datamodel=bigip-objectmodel-cert by host,all.devicegroup,all.facility | rename latest(all.) AS * all. AS * | join host cert_name [| tstats latest(all.cert_name) from datamodel=bigip-objectmodel-profile where all.profile_type="client-ssl" by host, all.devicegroup, all.facility, all.profile_name | rename latest(all.) AS * all. AS ] | join host profile_name [| tstats values(all.app), latest(all.tenant) from datamodel=bigip-objectmodel-virtual-profiles by host, all.devicegroup, all.facility, all.profile_name | rename latest(all.) AS * values(all.) as * all. AS ] | makemv delim=" " app | mvexpand app

| search tenant=tenant_a app=mail.clearshark.net | rename cert_expiration_date_human AS expires | eval days_remaining=round((cert_expiration_date-now())/(360024),0) | sort days_remaining | table facility,devicegroup,cert_name,CN,expires,days_remaining

All of my cert objects, ssl profile objects and virtual profile objects are being reported correctly into splunk. It seems this search though only returns the latest (hence the latest command) ssl cert object and joins all post objects in the search. It then searches for the requested app. Unfortunately, if the app isn't associated with this ssl profile, you do not get any results. I think instead of latest, values should be used with the mvexpand command. I've replaced the search with this

| tstats values(all.cert_name), values(all.cert_expiration_date), values(all.cert_expiration_date_human),values(all.CN) from datamodel=bigip-objectmodel-cert by host,all.devicegroup,all.facility | rename values(all.) AS * all. AS * | mvexpand cert_name | join host cert_name [| tstats values(all.cert_name) from datamodel=bigip-objectmodel-profile where all.profile_type="client-ssl" by host, all.devicegroup, all.facility, all.profile_name | rename values(all.) AS * all. AS ] | mvexpand profile_name | join host profile_name [| tstats values(all.app), values(all.tenant) from datamodel=bigip-objectmodel-virtual-profiles by host, all.devicegroup, all.facility, all.profile_name | rename values(all.) AS * values(all.) as * all. AS ] | makemv delim=" " app | mvexpand app

| search tenant=tenant_a app=mail.clearshark.net | rename cert_expiration_date_human AS expires | eval days_remaining=round((cert_expiration_date-now())/(360024),0) | sort days_remaining | table facility,devicegroup,cert_name,profile_name

The only thing I'm working on now is how to properly bring in the cn and expiration date. Anytime I expand those out, I get 100s of results. Any suggestions would be great!

[Jessicachi_3022]

Hello Ken,

Thank you so much for creating such a wonderful iAPP and splunk app. I would like to find out how I can turn off syslog information from being sent to splunk since it is consuming a lot of splunk data and we already have a separate syslog server. I tried to turn off the syslog feature from the iApp but it's telling that i can not perform the action because the vs/irule is being used. I also tried to disable the splunk-hec-syslog virtual server but that just prevent the F5 from sending any data to splunk. Do you think it's better to blacklist syslog information on splunk side? my 2nd question is regarding the healthscore calculation. I found that the caculation uses values such as app_device_uptime_health=1/0 but i could not figure out how you arrived at those values. could you please explain the process? thank you in advance!

[The-messenger]

Ken, thanks again for this iapp, very good! If installing on a VCMP host, that host will need a Self-IP configured, correct?

[jspiglerj2rsolves]

Has anyone else ran into these errors?

message from "python /opt/splunk/etc/apps/f5/bin/f5_kpi_summary_generator.py" application F5_KPI_Result=ERROR: [spl2.domain.net] Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.

Its affecting my KPI generation. Wanted to see if anyone else is having this issue.

[Shayza_312029]

Hi,

Any one may notice a bug when enabling "Role Based Access Controls"? Every time that I'm enabling it the LTM is losing the connection to Splunk (status 400), after disabling it the LTM seceded to establish the connection.

[jspiglerj2rsolves]

Figured out my issue

message from "python /opt/splunk/etc/apps/f5/bin/f5_kpi_summary_generator.py" application F5_KPI_Result=ERROR: [spl2.domain.net] Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.

Resource constraint from the CPU side of the house. datamodel summary searches were timing out because we didn't have enough cores allocated for the indexers.

Cheers!

[mwsmith87]

I am having issues with missing data anytime I look through any of the various dashboards or search for data. It says that there are duplicate tenant values causing a conflict. Anyone have any idea what should be done to correct that?