SSL Orchestrator Enhanced Uses Case: Remote Logging
Introduction This use case allows you to configure the BIG-IP SSL Orchestrator to send detailed logging to a remote Syslog server. Logging is an important aspect of SSL Orchestrator operation and troubleshooting. The volume of data created by debug logging is significant and should ideally be sent off-box for analysis and archiving. The following instructions demonstrate how to configure Remote Logging. Logging Level Logging verbosity is configured in the BIG-IP Configuration Utility. Under SSL Orchestrator select Configuration > Logs > Settings. Logging verbosity is set to Error by default. Change this to Debug for the Per-Request Policy and SSL Orchestrator Generic. Click Save when done. Note: For simplified logging that combines each connection flow into a single summary log, only enable SSL Orchestrator Generic at level Informational of higher. These log settings are Global and can be over ridden by per-Topology logging settings. Create a Pool for the Syslog server Under Local Traffic select Pools. Click Create. Give it a name, Remote_syslog_pool in this example. Give the Node a Name, syslog_server in this example. Enter the IP address of the syslog server and port 514 for the Service Port. Click Add. Note: 514 is the common port for syslogd but may be different in your environment. If desired, add a Health Monitor like gateway_icmp. Use the << to move it from Available to Active. Click Finished when done. Create Logging Destination Under SSL Orchestrator select Configuration > Logs > System. Then select Configuration > Log Destinations. Click Create. Give it a name, remote_syslog in this example. Select Remote High-Speed Log as the Type. Note: The Remote High-Speed Log (HSL) uses the data plane while Remote Syslog uses the management plane. HSL logging is preferred due to better, sustained performance. For more information on HSL click here. For Pool Name select the Pool created previously, Remote_syslog_pool in this example. Set the Protocol to UDP or TCP (typically UDP). Click Finished. Configure the Log Publisher From the same screen click Configuration > Log Publishers. Click on sys-sslo-publisher to edit it. Select the local-syslog and click the >> to move it to Available. Select the remote_syslog and click << to move it to Selected. Click Update when done. The configuration is now complete. Detailed logs should now be sent to your Syslog server. Verify it’s Working Check the Syslog Pool Statistics. From the BIG-IP Configuration Utility select Local Traffic > Pools. Select Statistics. If it is working you should see a non-zero value for Bits and Packets. Check your Syslog Server to verify it is receiving logs from SSL Orchestrator. In this example I’m running a packet capture on the Syslog Server to check that packets are being sent from the BIG-IP to the Syslog Server. In the example above you can see that the BIG-IP (10.0.0.1) is sending packets to the Syslog Server (10.0.0.2) on UDP port 514. You can also see the details of the Syslog message in the circle. Note: BIG-IP SSL Orchestrator needs a Self IP Address in order to send detailed logging to the Syslog Server. If deployed in Layer 2 mode you will need to configure a new Self IP Address. You cannot assign an IP address to an interface in an L2 vwire group. If deployed in Layer 3 mode you can use an existing Self IP Address as long as it can reach the Syslog Server. Ideally though, the Syslog traffic should not be on the same interface(s) as client/server traffic. In this example BIG-IP is configured with the Self IP 10.0.0.1 which is on the same subnet as the Syslog Server at IP address 10.0.0.2. Summary In this SSL Orchestrator Use Case you learned how to enable detailed logging on BIG-IP and have the logs sent to a remote Syslog Server.Load Balancing TCP TLS Encrypted Syslog Messages
Syslog messages sent via TCP are not always evenly distributed among backend syslog servers because multiple syslog messages can be sent in a single TCP connection. This article utilizes the F5 BIG-IP Generic Message Routing Framework (MRF) to evenly distribute syslog messages among backend syslog pool members. This solution also uses TLS to protect the confidentiality of the syslog messages. This article is based off the work done by Mark Lloyd in this DevCentral Technical Article: A Simple One-way Generic MRF Implementation to load balance syslog message. In his article, Mark explains how to setup Generic Message Routing Framework (MRF) to distribute syslog messages sent via TCP to a pool of syslog servers. This article adds the necessary configuration to TLS encrypt, decrypt, and re-encrypt the messages. This was tested on BIG-IP version 17.1.0.1. The first step is to define the message routing protocol. The difference between the default protocol (genericmsg) is the field no-response must be configured to yes if this is a one-way stream. Otherwise, the server side will allocate buffers for return traffic that will cause severe free memory depletion. Note: The message-terminator is set to "%0a", this represents a newline character in hex and is used to separate the syslog messages. This value can be changed if a different delineator is required. ltm message-routing generic protocol simple_syslog_protocol { app-service none defaults-from genericmsg description none disable-parser no max-egress-buffer 32768 max-message-size 32768 message-terminator %0a no-response yes } An iRule must be configured on both the Virtual Server and Generic Transport Config. This iRule must be linked as a profile in both the virtual server and the generic transport configuration. ltm rule mrf_simple { when CLIENT_ACCEPTED { GENERICMESSAGE::peer name "[IP::local_addr]:[TCP::local_port]_[IP::remote_addr]:[TCP::remote_port]" } when SERVER_CONNECTED { GENERICMESSAGE::peer name "[IP::local_addr]:[TCP::local_port]_[IP::remote_addr]:[TCP::remote_port]" } } The next item to configure is the generic transport config. The generic transport config has the generic protocol configured along with the iRule to setup the server-side peers. A server-side SSL profile is also configured here to TLS encrypt the traffic to the backend syslog servers. ltm message-routing generic transport-config simple_syslog_tcp_tc { ip-protocol tcp profiles { serverssl-insecure-compatible { } simple_syslog_protocol { } tcp { } } rules { mrf_simple } } Nodes are defined for the backend Syslog servers. ltm node 10.1.20.201 { address 10.1.20.201 } ltm node 10.1.20.202 { address 10.1.20.202 } A pool is created containing the nodes. ltm pool syslog_pool { members { 10.1.20.201:514 { address 10.1.20.201 } 10.1.20.202:514 { address 10.1.20.202 } } } The next step is to create the generic message routing peer. This peer is used to identify the pool of syslog servers that the syslog messages will be routed to. ltm message-routing generic peer simple_syslog_peer { pool syslog_pool transport-config simple_syslog_tcp_tc } Now that the peer is defined, a generic route can be created to send traffic to the peer. ltm message-routing generic route simple_syslog_route { peers { simple_syslog_peer } } A generic router is configured with the generic route. ltm message-routing generic router simple_syslog_router { app-service none defaults-from messagerouter description none ignore-client-port no max-pending-bytes 23768 max-pending-messages 64 mirror disabled mirrored-message-sweeper-interval 1000 routes { simple_syslog_route } traffic-group traffic-group-1 use-local-connection yes } A client-ssl profile is configured to associate the certificates to the Virtual Server. ltm profile client-ssl syslog-ng_client { app-service none cert-key-chain { syslog-ng-new_f5demos_ca_0 { cert syslog-ng-new chain f5demos_ca key syslog-ng-new } } defaults-from clientssl inherit-ca-certkeychain true inherit-certkeychain false } A virtual server is created to receive incoming TLS-encrypted TCP syslog messages. ltm virtual mrftest_simple { creation-time 2024-10-08:09:37:52 destination 10.1.10.101:514 ip-protocol tcp last-modified-time 2024-10-08:13:55:49 mask 255.255.255.255 profiles { simple_syslog_protocol { } simple_syslog_router { } syslog-ng_client { context clientside } tcp { } } rules { mrf_simple } serverssl-use-sni disabled source 0.0.0.0/0 source-address-translation { type automap } translate-address enabled translate-port enabled vs-index 2 } Conclusion This example is from a use case where a single syslog client was sending to a TCP load balancer. The load balancer was not evenly distributing the load among the backend servers because multiple messages were being sent as part of a single TCP connection. This solution utilizes a generic Message Routing Framework to evenly distribute TCP syslog messages. TLS encryption is also used on the client to load balancer connection as well as the load balancer to backend server connection to protect the confidentiality of the syslog messages.
Need to verify if F5 is sending logs to Syslog Server for local1 Facility
With the syslog setting as below how can i confirm it will send the APM related logs to remote syslog server ? This doc ( https://support.f5.com/csp/article/K15521451 ) says Local1 facility will carry APM related logs but below output does not show anything related to Local 1. Only shows Local6. Can you advise ? (/Common)(tmos)# list sys syslog all-properties sys syslog { auth-priv-from notice auth-priv-to emerg clustered-host-slot enabled clustered-message-slot disabled console-log enabled cron-from warning cron-to emerg daemon-from notice daemon-to emerg description none include none iso-date disabled kern-from debug kern-to emerg local6-from notice local6-to emerg mail-from notice mail-to emerg messages-from notice messages-to warning remote-servers { 10.8.11.11 { description none host 10.8.11.11 local-ip 10.8.22.22 remote-port 514 } } user-log-from notice user-log-to emerg
BigIP version 10 and logs to remote syslog server
Hi Guys, I have a bigip 3600 version 10 running. I configure the below command to send syslog to a remote serfer, yet I am not geting the logs on the syslog server. Checking the traffic on the network shows that the bigip is not sending syslog traffic. modify /sys syslog remote-servers add { SIEM { host 10.2.160.34 remote-port 514 }}
HTTP Payload logging breaks HTTP Keep-Alive
Afternoon all, I've written an iRule to record the request/response payload on a REST HTTP API. The rule looks like: when CLIENT_ACCEPTED { if { $static::PayloadLoggerDebug or $f5_connection_debug } { log local0.debug "Processing CLIENT_ACCEPTED." } Set the payload logging flag set log_payload 1 } when HTTP_REQUEST priority 800 { if { $static::PayloadLoggerDebug or $f5_connection_debug } { log local0.debug "$log_prefix: Processing HTTP_REQUEST at priority 800..." } Skip logging if no members available if {$splunk_bypass}{ if { $static::PayloadLoggerDebug or $f5_connection_debug } { log local0.debug "$log_prefix: Splunk HSL pool is down. Bypassing logging..." } return } Don't allow data to be chunked if { [HTTP::version] eq "1.1" } { if { [HTTP::header is_keepalive] } { HTTP::header replace "Connection" "Keep-Alive" } if { $static::PayloadLoggerDebug or $f5_connection_debug } { log local0.debug "$log_prefix: Switching to HTTP Version 1.0." } HTTP::version "1.0" } Split out request headers and munge into string set headers [HTTP::header names] if { $static::PayloadLoggerDebug or $f5_connection_debug } { log local0.debug "$log_prefix: Request HTTP Headers = $headers" } set request_headers "'" foreach header $headers { set value [HTTP::header value $header] if { $static::PayloadLoggerDebug or $f5_connection_debug } { log local0.debug "$log_prefix: Request HTTP Header $header value = $value" } append request_headers "$header=$value " } set request_headers [string trimright $request_headers " "] append request_headers "'" } when HTTP_REQUEST_DATA { if { $static::PayloadLoggerDebug or $f5_connection_debug }{ log local0.debug "$log_prefix: Collected [HTTP::payload length] bytes."} set request_payload [HTTP::payload] } when HTTP_RESPONSE priority 50 { if { $static::PayloadLoggerDebug or $f5_connection_debug } { log local0.debug "$log_prefix: Processing HTTP_RESPONSE at priority 50..." } Skip logging if no members available if {$splunk_bypass}{ if { $static::PayloadLoggerDebug or $f5_connection_debug } { log local0.debug "$log_prefix: Splunk HSL pool is down. Bypassing logging..." } return } Split out the response headers and munge into string set headers [HTTP::header names] if { $static::PayloadLoggerDebug or $f5_connection_debug } { log local0.debug "$log_prefix: Response HTTP Headers = $headers" } set response_headers "'" foreach header $headers { set value [HTTP::header value $header] if { $static::PayloadLoggerDebug or $f5_connection_debug } { log local0.debug "$log_prefix: Response HTTP Header $header value = $value" } append response_headers "$header=$value " } set response_headers [string trimright $response_headers " "] append response_headers "'" Collect the response if { $response_length > 0 } { if { $static::PayloadLoggerDebug or $f5_connection_debug } { log local0.debug "$log_prefix: Collecting $response_length bytes from response." } HTTP::collect $response_length } Calculate actual content-length set response_length_real [HTTP::payload length] if { $static::PayloadLoggerDebug or $f5_connection_debug } { log local0.debug "$log_prefix: Real response content-length = $response_length_real." } Correct the response_length to correct value if required. if { $response_length != $response_length_real } { if { $static::PayloadLoggerDebug or $f5_connection_debug } { log local0.debug "$log_prefix: Updated \$response_length value." } set response_length $response_length_real if { $static::PayloadLoggerDebug or $f5_connection_debug } { log local0.debug "$log_prefix: New \$response_length value is $response_length." } } } when HTTP_RESPONSE_DATA { if { $static::PayloadLoggerDebug or $f5_connection_debug } { log local0.debug "$log_prefix: Processing HTTP_RESPONSE_DATA." } Gather response data set response_payload "[HTTP::payload]" } A lot of the above feeds into a larger iRule framework. E.g. we use a SplunkHTTPS iRule to actually do the HSL logging out to a syslog server. The 'HTTP::collect' is also called from that iRule aswell. However when testing this iRule, I've identified an issue whereby it appears to be breaking HTTP Keep-alive connections to some of our GF3 application servers. Removing this rule restores the keep-alive functionality. Any pointers on how I can maintain the keep-alive functionality and also be able to log the request/response data? Cheers Gavin
