Most efficient methods for Connection logging?
Does anyone have real world experience with logging connections at a high rate? If so, which methods are you using to collect and transmit the data? We have a requirement to log all connections going through our F5 devices. Things like the client/server-side IPs/ports as well as HTTP details for HTTP VIPs and DNS details from our GTMs. It's the Whitehouse M-21-31 mandate if anyone if familiar with it. I've used Request Logging Profiles and various iRules with HSL to collect this type of data before, but I've never been too concerned about overhead because I would only apply them as needed, like when t-shooting an issue with a VIP. Our busiest appliance pushes around 150k conn/sec and 5k HTTP req/sec, so I now have consider the most efficient methods to avoid any kind of impact to traffic flows. I've done some lab testing with several different methods but I can't do any meaningful load tests in that environment.Below are some of my opinions based on my lab testing so far. Data Collection AVR - I like that this single feature can meet all the requirements for collecting TCP, HTTP, and DNS data. It would also be relatively easy to perform audits to ensure the VIPs have the necessary Analytics profiles as we can manage it from the AVR profiles themselves. My main concern is the overhead that results from the traffic analysis. I assume it has to maintain a large database where it stores all the analyzed data even if we just ship it off to Splunk. Even the data shipped off to Splunk includes several different logs for each connection (each with a different 'Entity'). Request Logging Profile- This is fairly flexible and should have low overhead since the F5 doesn't need to analyze any of the data like AVR does. This only collects HTTP data so we still need another solution to collect details for non HTTP VIPs. It would be a pain to audit since we don't have use any kind of deployment templates or automation. iRule - This provides a lot of flexibility and it is capable of collecting all the necessary data, but I don't know how well performance overhead compares to AVR. This would also be a pain to audit due to lack of deployment templates and automation. Data Transmission HSL UDP Syslog- I imagine this is the most efficient method to send events, but it's likely only a matter of time before we are required to use TCP/TLS. Telemetry Streaming - This is the more modern method and it offers some interesting features like System Poller, which could eventually allow us to move away from SNMP polling. We would need a workaround for our GTM-only devices because they cannot run a TS listener.
AVR average server latency istats with Irule
I am looking for a particular use case to be able to read the AVR analytics average server response time and be able to mark the specific server down and use this information to make a proper load balancing decision. So basically lets say when the average server response latency is higher then 1000ms then we should not send traffic there but do a load balancing reselection. The following command is used in TMSH to view this data. Now I would like to collect this within the Irule. show analytics http report view-by pool-member range now-3d measures {average-server-latency } In the analytics profile I have seen the option "publish Irule statistics" and that is what I need, but how do I read the specific metric data from within the Irule? I read about istats is that what I need?
AVR Profile and Remote syslog
Hi all, Just wondering if anyone has managed to use an analytics profile to export the logs/data to an external source, ideally rsyslog? There was a lot of reading about SPLUNK and iApp's and also setting up log publishers/destinations, but I could not find anyone having success logging to a remote syslog server direct from an analytics profile. I think this is possible from the manual of AVR. Any help would be great, Thanks.AVR remote logging HSL pool
Dear All, I am trying to set up AVR remote logging to send web statistics to an external syslog server, but no traffic is being send at all. I configured the pool for sending syslog traffic. Configured ARV Analytics profile for external statistics logging type enabled. The log publisher named syslog used in the Analytics profile refers to the Syslog-ECM log destination. The syslog-ECM log destination is configured as type syslog and refers to the syslog pool. But traffic is not being send. If I use an Irule for sending logs to the destination it does work, it is not an IP connectivity issue but a more AVR specific one. Someone tried this out before, please leave your comments. Thanks
Analytics : HTTP : Transactions - Export Top URL
Hello, I'm using AVR (Application Visibility Reporting)for my F5 Big IP LTM traffic. When i'm going to my statistics, I can see 29 975 total entries for my URL. But When I try to export all of these URLs in CSV format, my CSV file contains only the first 10 entries. Do I need to apply a particular method to export all of my 29,975 entries to the same CSV file to process this data? Thanks for your help, Hugo
F5 LTM+AVR
Hello, I'm trying to figure out if it's possible to collect source IP addresses (client's connections) via F5 AVR provisioned with LTM. GTM is running on a separate device. So far I haven't been able to find how to do it, even after checking the box for client ip addresses (in avr settings), etc. Anyone familiar/have an idea? I couldn't find much on devcentral or Google for it. Thanks!
