Intermediate iRules: High Speed Logging - Spray Those Log Statements!
High Speed Logging has been around since version 10.1, and has been integral to many projects over the past few years. Prior to HSL's introduction, logging remotely was configured entirely in syslog or could be handled in iRules by specifying a destination in the log statement. One enhancement with HSL to that scenario was to allow a pool of servers to be configured for a destination, so given a pool of servers, the log messages were sure to arrive somewhere (ok, for TCP they were sure to arrive!) A drawback with either the log or HSL::send command, however, is that the message was only going to hit one destination. A workaround for that problem is to just use as many commands as necessary to hit all your destinations, but that's not very efficient. Enter the publisher. Beginning in version 11.3, a new option to the HSL::open command was added that allows you to send data to a log publisher instead of only to a pool. This allows you to spray that data to as many servers as you like. In my test setup, I used alias interfaces on a linux virtual machine as the destinations, and created a pool for each to be added to the publisher: ltm pool lp1 { members { 192.168.101.20:514 { address 192.168.101.20 } } } ltm pool lp2 { members { 192.168.101.21:514 { address 192.168.101.21 } } } ltm pool lp3 { members { 192.168.101.22:514 { address 192.168.101.22 } } } Once I have the pools defined, I create the log destinations: sys log-config destination remote-high-speed-log lp1 { pool-name lp1 protocol udp } sys log-config destination remote-high-speed-log lp2 { pool-name lp2 protocol udp } sys log-config destination remote-high-speed-log lp3 { pool-name lp3 protocol udp } Finally, I create the publisher for use in the iRules: sys log-config publisher lpAll { destinations { lp1 lp2 lp3 } } That's all the background magic required to get to the iRules showing off the -publisher option in HSL::open: ltm rule testrule { when CLIENT_ACCEPTED { set lpAll [HSL::open -publisher /Common/lpAll] } when HTTP_REQUEST { HSL::send $lpAll "<190> [IP::client_addr]:[TCP::client_port]-[IP::local_addr]:[TCP::local_port]; [HTTP::host][HTTP::uri]" } } Finally, some visual evidence for the skeptics out there: You can see that all three destinations got the message, and the message arrived as formatted. So now, armed with this new option (as of version 11.3), go forth and code!
Apache Style Logging with HSL
Problem this snippet solves: When SNATing to servers, the client IP is lost. This was information our security group and developers wanted to have available, so I created an iRule to use the HSL functionality in BigIP 10 to do that. Without cracking open the HTTP stream, I was unable to get the REMOTE_USER CGI variable, so I used that field to log the HTTP::host instead (since that information is lost with several VIPs logging to the same location). Code : # Apache: ClientIP - Username 10/Oct/2000 # iRule: ClientIP - Host 10/Oct/2000 when CLIENT_ACCEPTED { set hsl [HSL::open -proto UDP -pool syslog.example.com_syslog_pool] set clientip [IP::remote_addr] } when HTTP_REQUEST { set method [HTTP::method] set uri [HTTP::uri] set host [HTTP::host] } when HTTP_RESPONSE { set now [clock format [clock seconds] -format "%d/%b/%Y:%H:%M:%S %z"] set contentlength [HTTP::header "Content-Length"] # Log HTTP request via syslog protocol as local7.info; see RFC 3164 for more info # local7 = 23; info = 6; 8*23 + 6 = 190 HSL::send $hsl "<190> $clientip $host - $now \"$method $uri HTTP/[HTTP::version]\" [HTTP::status] $contentlength\n" #log local0. "<190> $clientip $host - $now \"$method $uri HTTP/[HTTP::version]\" [HTTP::status] $contentlength" } Tested this on version: 10.2
SSL Orchestrator Enhanced Uses Case: Remote Logging
Introduction This use case allows you to configure the BIG-IP SSL Orchestrator to send detailed logging to a remote Syslog server.Logging is an important aspect of SSL Orchestrator operation and troubleshooting.The volume of data created by debug logging is significant and should ideally be sent off-box for analysis and archiving.The following instructions demonstrate how to configure Remote Logging. Logging Level Logging verbosity is configured in the BIG-IP Configuration Utility.Under SSL Orchestrator select Configuration > Logs > Settings. Logging verbosity is set to Error by default.Change this to Debug for the Per-Request Policy and SSL Orchestrator Generic.Click Save when done. Note: For simplified logging that combines each connection flow into a single summary log, only enable SSL Orchestrator Generic at level Informational of higher.These log settings are Global and can be over ridden by per-Topology logging settings. Create a Pool for the Syslog server Under Local Traffic select Pools. Click Create. Give it a name, Remote_syslog_pool in this example.Give the Node a Name, syslog_server in this example.Enter the IP address of the syslog server and port 514 for the Service Port.Click Add. Note: 514 is the common port for syslogd but may be different in your environment. If desired, add a Health Monitor like gateway_icmp.Use the << to move it from Available to Active. Click Finished when done. Create Logging Destination Under SSL Orchestrator select Configuration > Logs > System. Then select Configuration > Log Destinations. Click Create. Give it a name, remote_syslog in this example.Select Remote High-Speed Log as the Type. Note: The Remote High-Speed Log (HSL) uses the data plane while Remote Syslog uses the management plane.HSL logging is preferred due to better, sustained performance.For more information on HSL click here. For Pool Name select the Pool created previously, Remote_syslog_pool in this example. Set the Protocol to UDP or TCP (typically UDP).Click Finished. Configure the Log Publisher From the same screen click Configuration > Log Publishers. Click on sys-sslo-publisher to edit it. Select the local-syslog and click the >> to move it to Available. Select the remote_syslog and click << to move it to Selected. Click Update when done. The configuration is now complete.Detailed logs should now be sent to your Syslog server. Verify it’s Working Check the Syslog Pool Statistics.From the BIG-IP Configuration Utility select Local Traffic > Pools. Select Statistics. If it is working you should see a non-zero value for Bits and Packets. Check your Syslog Server to verify it is receiving logs from SSL Orchestrator.In this example I’m running a packet capture on the Syslog Server to check that packets are being sent from the BIG-IP to the Syslog Server. In the example above you can see that the BIG-IP (10.0.0.1) is sending packets to the Syslog Server (10.0.0.2) on UDP port 514.You can also see the details of the Syslog message in the circle. Note: BIG-IP SSL Orchestrator needs a Self IP Address in order to send detailed logging to the Syslog Server.If deployed in Layer 2 mode you will need to configure a new Self IP Address.You cannot assign an IP address to an interface in an L2 vwire group.If deployed in Layer 3 mode you can use an existing Self IP Address as long as it can reach the Syslog Server.Ideally though, the Syslog traffic should not be on the same interface(s) as client/server traffic.In this example BIG-IP is configured with the Self IP 10.0.0.1 which is on the same subnet as the Syslog Server at IP address 10.0.0.2. Summary In this SSL Orchestrator Use Case you learned how to enable detailed logging on BIG-IP and have the logs sent to a remote Syslog Server.Device Fingerprinting for mobile devices
We have published Microsoft Exchange server behind ASM+APM policies. I am using BotDefense to generate Device IDs for the connecting clients and then inserting the value of "device_id" variable into HTTP header and then passing it to APM. APM extracts the "device_id" value from HTTP header and sets it as an APM variable: ACCESS::session data set "session.custom.device_id" "$device_id" This APM session variable is then called in APM iRules to perform the required logic. So far, the logic and traffic flow is working as expected. Now, here comes the problem part: If the client is not a web browser, the whole logic fails because BotDefense is generating Device IDs based on JavaScript challenge. In case of Microsoft Exchange, the client could be a mobile device using the native email app of the phone (not mobile browser) like the mail app of Andriod or iOS devices. How can we perform device fingerprinting and generate Device IDs in such case? All mobile devices use the standard Exchange protocol "Microsoft ActiveSync" which is based on HTTPS. When traffic hits my virtual server from a mobile client, BIG-IP can detect it and differentiate it from other web browser traffic because all requests coming from mobile devices have this particular URI string in the HTTP Request: "/Microsoft-Server-ActiveSync". But because it is not a web browser, the JavaScript challenge is not performed and no Device ID generated. My question is: How can we perform fingerprinting and generate Device IDs for mobile devices (not mobile browsers)? Here is my ASM iRule which is handling the Device IDs and fingerprinting: ============================================================================================================================== when RULE_INIT { set static::TPS_Value 1 set static::debug 1 # set as 1 - send request logs set 0 if no request logs should be sent. set static::PBD_debug 1 # list of botdefense actions you want to get request log on set static::Logged_PBD_actions "tcp_rst browser_challenge internal_bigip_response captcha_challenge" set static::host_header "Host: webmail.company.com" } when HTTP_REQUEST { set hsl [HSL::open -proto TCP -pool ASM_Log_Pool2] set http_request [HTTP::request] #HSL::send $hsl $http_request } when BOTDEFENSE_REQUEST { #for demo purpose, make the challange valid from the first request - make sure you go to the default if {[HTTP::uri] equals "/"} { BOTDEFENSE::cs_allowed true } #Mandate the device_id attribute extraction BOTDEFENSE::cs_attribute device_id enable } when BOTDEFENSE_ACTION { set device_id [BOTDEFENSE::device_id] if {$static::debug > 0} {log "reason is, [BOTDEFENSE::reason], action is [BOTDEFENSE::action], botdefense device_id is: $device_id"} if {$static::Logged_PBD_actions contains [BOTDEFENSE::action]} { set botdefense_action [BOTDEFENSE::action] set botdefense_reason [BOTDEFENSE::reason] set PBD_header [concat Host: webmail.company.com\r\nPbd_Action: $botdefense_action\r\nPbd_reason: $botdefense_reason] set asm_http_requet_log [string map -nocase [list $static::host_header $PBD_header] $http_request] if {($static::PBD_debug > 0) && ([info exists asm_http_requet_log])} { HSL::send $hsl $asm_http_requet_log } } log "action is [BOTDEFENSE::action], reason is: [BOTDEFENSE::reason] cs_allowed is: [BOTDEFENSE::cs_allowed]" if {([BOTDEFENSE::action] eq "tcp_rst") && [BOTDEFENSE::cs_allowed] eq 0} { set res [BOTDEFENSE::action custom_response { sorry i am blocking you, try to restart the session } 200] if {$res eq "ok"} { set botdefense_responded 1 } } #if {[BOTDEFENSE::action] eq "captcha_challenge"} { #set res [BOTDEFENSE::action allow] #log "captcha challange with res $res" #if {$res eq "ok"} { #log "bypass allow" #} #} #} when ASM_REQUEST_DONE { if {$static::debug > 0} {log "http uri is [HTTP::uri]"} virtual Hackazone_APM_virt } when HTTP_REQUEST_SEND { clientside { # Need to force the host header replacement and HTTP:: commands into the clientside context # as the HTTP_REQUEST_SEND event is in the serverside context if {$static::debug > 0} {log "device id is: $device_id"} HTTP::header insert "device_id" "$device_id" #if { $suspicious_browser eq "1" } { #HTTP::header insert "suspicious_browser" "1" #log "sending suspicious_browser header" #} #log "after the change [HTTP::request]" } } when HTTP_RESPONSE_RELEASE { if {[info exists botdefense_responded]} { HTTP::header insert "X-TS-BP-Action" "2" } } =================================================================================================================================== Many thanks.
BigIP DNS Log queries
Hello We have GTM only licensed VM And we'd like to log all the queries to either the local system to remote syslog But neither ways are working Not sure if this matters but I'd like to mention that we're using GTM as cache forwarded zone, and if no domain match there then it falls to the default pool attached to the listener So back to logging, we tried two ways: local-db--publisher , but not sure where to find all queries logs?... remote hsl to to kiwi server but also there no logs been sent I simply what to see queries logs ... how should i accomplish this task?
Log large HTTP payloads in chunks locally and remotely
Problem this snippet solves: Log HTTP POST request payloads remotely via High Speed Logging (HSL) to a syslog server and locally. Code : # Log POST request payloads remotely via HSL to a syslog server and locally. # Based on Steve Hillier's example and the HTTP::collect wiki page # https://devcentral.f5.com/s/wiki/iRules.http__collect.ashx # Note that although any size payload can theoretically be collected, the maximum size of a Tcl variable in v9 and v10 is 4MB # with a smaller functional maximum after charset expansion of approximately 1Mb. # In v11, the maximum variable size was increased to 32Mb. when RULE_INIT { # Log debug to /var/log/ltm? 1=yes, 0=no set static::payload_dbg 1 # Limit payload collection to 5Mb set static::max_collect_len 5242880 # HSL pool name set static::hsl_pool "my_hsl_tcp_pool" # Max characters to log locally (must be less than 1024 bytes) # https://devcentral.f5.com/s/wiki/iRules.log.ashx set static::max_chars 900 } when HTTP_REQUEST { # Only collect POST request payloads if {[HTTP::method] equals "POST"}{ if {$static::payload_dbg}{log local0. "POST request"} # Open HSL connection set hsl [HSL::open -proto TCP -pool $static::hsl_pool] # Get the content length so we can request the data to be processed in the HTTP_REQUEST_DATA event. if {[HTTP::header exists "Content-Length"]}{ set content_length [HTTP::header "Content-Length"] } else { set content_length 0 } # content_length of 0 indicates chunked data (of unknown size) if {$content_length > 0 && $content_length < $static::max_collect_len}{ set collect_length $content_length } else { set collect_length $static::max_collect_len } if {$static::payload_dbg}{log local0. "Content-Length: $content_length, Collect length: $collect_length"} } } when HTTP_REQUEST_DATA { # Log the bytes collected if {$static::payload_dbg}{log local0. "Collected [HTTP::payload length] bytes"} # Send all the collected payload to the remote syslog server HSL::send $hsl "<190>[HTTP::payload]\n" # Log the payload locally if {[HTTP::payload length] < $static::max_chars}{ log local0. "Payload=[HTTP::payload]" } else { # Initialize variables set remaining $payload set position 0 set count 1 set bytes_logged 0 # Loop through and log each chunk of the payload while {[string length $remaining] > $static::max_chars}{ # Get the current chunk to log (subtract 1 from the end as string range is 0 indexed) set current [string range $remaining $position [expr {$position + $static::max_chars -1}]] log local0. "chunk $count=$current" # Add the length of the current chunk to the position for the next chunk incr position [string length $current] # Get the next chunk to log set remaining [string range $remaining $position end] incr count incr bytes_logged $position log local0. "remaining bytes=[string length $remaining], \$position=$position, \$count=$count, \$bytes_logged=$bytes_logged" } if {[string length $remaining]}{ log local0. "chunk $count=$current" incr bytes_logged [string length $remaining] } log local0. "Logged $count chunks for a total of $bytes_logged bytes" } }
Most efficient methods for Connection logging?
Does anyone have real world experience with logging connections at a high rate? If so, which methods are you using to collect and transmit the data? We have a requirement to log all connections going through our F5 devices. Things like the client/server-side IPs/ports as well as HTTP details for HTTP VIPs and DNS details from our GTMs. It's the Whitehouse M-21-31 mandate if anyone if familiar with it. I've used Request Logging Profiles and various iRules with HSL to collect this type of data before, but I've never been too concerned about overhead because I would only apply them as needed, like when t-shooting an issue with a VIP. Our busiest appliance pushes around 150k conn/sec and 5k HTTP req/sec, so I now have consider the most efficient methods to avoid any kind of impact to traffic flows. I've done some lab testing with several different methods but I can't do any meaningful load tests in that environment.Below are some of my opinions based on my lab testing so far. Data Collection AVR - I like that this single feature can meet all the requirements for collecting TCP, HTTP, and DNS data. It would also be relatively easy to perform audits to ensure the VIPs have the necessary Analytics profiles as we can manage it from the AVR profiles themselves. My main concern is the overhead that results from the traffic analysis. I assume it has to maintain a large database where it stores all the analyzed data even if we just ship it off to Splunk. Even the data shipped off to Splunk includes several different logs for each connection (each with a different 'Entity'). Request Logging Profile- This is fairly flexible and should have low overhead since the F5 doesn't need to analyze any of the data like AVR does. This only collects HTTP data so we still need another solution to collect details for non HTTP VIPs. It would be a pain to audit since we don't have use any kind of deployment templates or automation. iRule - This provides a lot of flexibility and it is capable of collecting all the necessary data, but I don't know how well performance overhead compares to AVR. This would also be a pain to audit due to lack of deployment templates and automation. Data Transmission HSL UDP Syslog- I imagine this is the most efficient method to send events, but it's likely only a matter of time before we are required to use TCP/TLS. Telemetry Streaming - This is the more modern method and it offers some interesting features like System Poller, which could eventually allow us to move away from SNMP polling. We would need a workaround for our GTM-only devices because they cannot run a TS listener.
alertd high cpu usage
Hello, So we tried taxing the HSL logging of our 12.1.2 cluster with several (4) simple while true; do curl http://virtual-ip/; done curl loops. The virtual IP had a simple iRule that logged [HTTP::request] several times. Our logging is BSD syslog to HSL, to a logstash pool. During our testing, we saw alertd rising to 100% CPU and maxing out there. CPU usage on the dashboard increased aswell, as you see in the image below. sys log-config destination remote-high-speed-log elk-hsl-destination { pool-name syslog-pool } sys log-config destination remote-syslog rsyslog-to-hsl-elk { remote-high-speed-log elk-hsl-destination } sys log-config filter elk-hsl-filter { level info publisher elk-hsl-publisher } sys log-config publisher elk-hsl-publisher { destinations { rsyslog-to-hsl-elk { } } } Any idea how we can combat this? We would like to use HSL to reduce CPU consumption, but this seems like a lot of fuss for simple logging. Ideas? Thanks!
