High Speed Logging vs Telemetry Streaming (Logging to SIEM)
My goals is to send web application traffic logs from my virtual servers to an external SIEM. It looks like there are quite a few ways to approach this, so I want to check with the community to see what works best for you. Ideally, this would be a high-volume configuration with logging enabled for 400+ public virtual servers. At a minimum, I would like to collect the client IP, user agent, URI path, virtual IP, virtual server name, pool, server name, and server-side response code. I reviewed the overview here: Getting Started with iRules: Logging & Comments | DevCentral It is clear that High Speed Logging (HSL) would be the preferred approach to ensure the resource and capacity burden is placed on the TMM (data plane) and not the control/management plane and to avoid writing to disk on the F5 BIG-IP host. I could write to syslog servers and forward these logs to my SIEM. The HSL seems to be straight forward to configure with a sample iRule looking like: when CLIENT_ACCEPTED { set vs [IP::local_addr]:[TCP::local_port] # Open HSL connections with configured pools set hsl_pool_1 [HSL::open -proto UDP -pool Pool_Syslog_1] set hsl_pool_2 [HSL::open -proto UDP -pool Pool_Syslog_2] } when SERVER_CONNECTED { set client [IP::client_addr]:[TCP::client_port] set srv [IP::remote_addr]:[TCP::remote_port] set log_message "<134>Client: $client connected to $vs and routed to server $srv at [clock format [clock seconds] -format \"%Y-%m-%d %H:%M:%S\"]" # Send logs to both HSL pools HSL::send $hsl_pool_1 $log_message HSL::send $hsl_pool_2 $log_message } However, when I searched through the DevCentral forums for references to SIEM logging, I found that most examples used Telemetry Streaming and AS3 for configuration. An official KB can be found here: Configure Azure sentinel or other telemetry consumer integration with BIG-IP and there is a f5devcentral github publication with configuration declarations for this approach for multiple SIEM vendors (e.g., analytics-vendor-dashboards/elastic at main · f5devcentral/analytics-vendor-dashboards for Elastic) For a use case like mine that involves high-volume logging in BIG-IP, do you know if HSL or Telemetry Streaming would be best to minimize the impact on BIG-IP?Logging Server-IP with OneConnect
Due to the usage of SNAT (design requirement) the servers are loosing the visibility, which real clients are connecting. Therefor the LTM should provide these information with internal logging (client-IP <-> VS <-> server-IP). But we are also using OneConnect, where logging of all sub-sequent requests of an existing server-side connection are getting "lost". Is there any solution available to still provide such a full logging information with OneConnect enabled? Or is disabling OneConnect the only solution? Thank you! Regards Stefan :)
Log message code list?
SOL16197: Reviewing BIG-IP log files describes local traffic log message format as: Message code is split into: Message code: The code that is associated with the message. The code is comprised of the following sub-codes: Product Code: The first two hex digits form the product code. For example, 0x01 is the BIG-IP product code. Subset Code: The third and fourth hex digits are the subset code. For example, 0x2a is the subset code for LIBHAL. Message Number: The next four digits form the message number within a module. Severity Level: The last digit between the colon symbols is the severity level, with 0 being the highest severity level. Are the Product and Subset codes listed anywhere? Would help in processing log messages further in Splunk or similar tool.
Logging Configuration in LTM HA
Hello everyone I'm doing a logging lab and I'm asking for your help to understand some things. I have two BIG-IP LTM in HA and a Qradar logging server. I have configured the Qradar as syslog servers at each HA node. At the Qradar level I receive the active logs but the standby only sends errors like "BIGIP_TMM_TIMMERR_PMBR_BACK_UP. I'd like to know if it's normal for the standby to send only error messages, and in general I'd like to understand how logging works in HA and what type of event each device sends to the server. thanks in advanceF5 APM Webtop - RDP Session Logging
Hello F5 Experts, I am relatively new to the F5 advanced ecosystem, am trying to generate useful logs from our APM Webtop environment and am hoping that someone can point me in the correct direction. I am trying to log the following things from our environment: Initial login's to the Webtop including ClientIP, Webtop portal address, Browser UserAgent, Client Username. (Optional) Client group membership/published resources when they log into Webtop. When a client opens a Web Portal Access from within Webtop including, ClientIP, Webtop portal address, Browser UserAgent, Client Username, Web Portal Access Address. (It would be good to get their session duration for this but that might not be fesable). When a client opens a RDP link from within the Webtop including, ClientIP, Webtop portal address, Browser UserAgent, Client Username, RDP Address, SessionCookie(?). When a client connects to a RDP session though the Webtop using one of the downloaded links, ClientIP, Webtop portal address, Client Username, RDP Address, SessionCookie(?), Session start and end time (Maybe two different log events?). From what I can tell this is likely to be an iRule. I think I have an idea how to do the Webtop portal logging, but what is really eluding me is how to log the RDP session connection and duration. Any help or a direction where to look would be greatly appreciated. Thank you,APM - How to configure logging of snat addresses for network access and app tunnels
Hello everyone, we are using BIG-IP Access Policy Manager to enable administrative access to systems via App Tunnel and Network Access resources. For security reasons, we need to be able to map requests logged on backend resources/systems (e.g. in SSH audit logs) to the session or user accessing said backend resource via App Tunnel or Network Access in APM. Currently, the following request information is logged. Network Access: May 17 14:42:00 tmm0 tmm[22565]: 01580002:5: /APM/ap_rmgw:Common:c1237463: allow ACL: #app_tunnel_/APM/Some_App-Tunnel@c1237463:15 packet: tcp 192.168.12.18:58680 -> 10.0.0.1:22 App Tunnels: May 17 14:41:10 tmm1 tmm1[22565]: 01580002:5: /APM/ap_rmgw:Common:c6787463: allow ACL: #app_tunnel_/APM/Some_App-Tunnel@c6787463:0 packet: tcp 89.229.152.144:63252 -> 10.0.0.1:2 For Network Access requests, an IP address of the lease pool configured in the Network Access resource is logged as the client IP. For App Tunnel requests, the public IP of the client accessing APM is logged as the client IP. In our setup, both requests will be NATed by APM before hitting the target system (through a snat pool in case of a Network Access request, through the active appliances backend IP in case of App Tunnels). Therefore, the APM self IPs (snat pool/appliance backend) will be logged on the target host, leading to us not being able to correlate logs in APM with logs on the target systems. Is there any way to log the SNAT/NAT addresses and ports used to access target systems through APM? I've tried using ACCESS_ACL_ALLOWED in an iRule to log additional information, unfortunately this event only seems to trigger on Portal Access resources, not when using App Tunnels or Network Access resources. Thank you, Fabian
Most efficient methods for Connection logging?
Does anyone have real world experience with logging connections at a high rate? If so, which methods are you using to collect and transmit the data? We have a requirement to log all connections going through our F5 devices. Things like the client/server-side IPs/ports as well as HTTP details for HTTP VIPs and DNS details from our GTMs. It's the Whitehouse M-21-31 mandate if anyone if familiar with it. I've used Request Logging Profiles and various iRules with HSL to collect this type of data before, but I've never been too concerned about overhead because I would only apply them as needed, like when t-shooting an issue with a VIP. Our busiest appliance pushes around 150k conn/sec and 5k HTTP req/sec, so I now have consider the most efficient methods to avoid any kind of impact to traffic flows. I've done some lab testing with several different methods but I can't do any meaningful load tests in that environment. Below are some of my opinions based on my lab testing so far. Data Collection AVR - I like that this single feature can meet all the requirements for collecting TCP, HTTP, and DNS data. It would also be relatively easy to perform audits to ensure the VIPs have the necessary Analytics profiles as we can manage it from the AVR profiles themselves. My main concern is the overhead that results from the traffic analysis. I assume it has to maintain a large database where it stores all the analyzed data even if we just ship it off to Splunk. Even the data shipped off to Splunk includes several different logs for each connection (each with a different 'Entity'). Request Logging Profile - This is fairly flexible and should have low overhead since the F5 doesn't need to analyze any of the data like AVR does. This only collects HTTP data so we still need another solution to collect details for non HTTP VIPs. It would be a pain to audit since we don't have use any kind of deployment templates or automation. iRule - This provides a lot of flexibility and it is capable of collecting all the necessary data, but I don't know how well performance overhead compares to AVR. This would also be a pain to audit due to lack of deployment templates and automation. Data Transmission HSL UDP Syslog - I imagine this is the most efficient method to send events, but it's likely only a matter of time before we are required to use TCP/TLS. Telemetry Streaming - This is the more modern method and it offers some interesting features like System Poller, which could eventually allow us to move away from SNMP polling. We would need a workaround for our GTM-only devices because they cannot run a TS listener.
Attaching w3c iRule to VS
Hi, I'm in midst of preparation to attach w3c iRule to all the VS in my internet facing BIGIP. I would be attaching to all the VS with http profile. But just wondering, if there would be any impact on the configuration (or change in Properties of VS) when I attach the iRule in question to the VS. Please confirm. Thanks, MSK
