Forwarding Logs to SIEM Tools via HTTP Proxy for F5 Distributed Cloud Global Log Receiver
Purpose This guide provides a solution for forwarding logs to SIEM tools that support syslog but lack HTTP/HTTPS ingestion capabilities. It covers the deployment and tuning of an HTTP Proxy log receiver configured to work with F5 Distributed Cloud (XC) Global Log Receiver settings. Audience: This guide is intended for technical professionals, including SecOps teams and Solution Architects, who are responsible for integrating SIEM tools with F5 XC Global Log Receiver. Readers should have a solid understanding of HTTP communication (methods, request body, reverse proxy), syslog, and data center network architecture. Familiarity with F5 XC concepts such as namespaces, log types, events, and XC-GLR is also required. Introduction: Problem Statement: SIEM tools often support syslog ingestion but lack HTTP/HTTPS log reception capabilities. Objective: Explain how to deploy and configure an HTTP Proxy to forward logs to F5 Distributed Cloud Global Log Receiver. Solution Overview: Architecture Diagram and workflow: Configuration Steps: Configure Global Log Receiver in F5 Distributed Cloud Console Navigate to: Home → Shared Configuration → Global Log Receiver Create or edit the Global Log Receiver settings for HTTP receiver Ensure the Global Log Receiver batch size is based on the payload size expected from F5 NGINX. Example configuration snap: Set Up NGINX as an HTTP Log Receiver Install NGINX on your designated server. Configure log_format Configure NGINX to accept HTTP POST requests only and forward access logs to syslog Example configuration snippet: log_format custom_log_format_1 escape=json $request_body; # Example: include request body only server { listen 443 ssl; server_name <logreceiver_server_name>; ssl_certificate /etc/ssl/<logreceiver_server_cert>; ssl_certificate_key /etc/ssl/<logreceiver_server_key>; # Other SSL/TLS configurations (e.g., protocols, ciphers) ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers HIGH:!aNULL:!MD5; client_body_in_single_buffer on; # The directive is recommended when using the $request_body variable, to save the number of copy operations involved client_body_in_file_only off; #default client_max_body_size 32M; # based on tuning gzip on; location /log_endpoint { # Allow only POST requests for sending log data limit_except POST { deny all; } # Configure access_log to write incoming data to a file # access_log /var/log/nginx/log_receiver.log custom_log_format_1; access_log syslog:server=127.0.0.1:514,facility=local7,tag=nginx,severity=info custom_log_format_1; proxy_pass http://localhost:8091/; # This dummy Internal server required to collect request_body variable. } } # dummy internal server to respond back 200 ok server { listen 8091; server_name localhost; location / { return 200 "Log received successfully."; } } Set Up rsyslog server Install/configure rsyslog on your designated server. Configure 60-nginx.conf file in /etc/rsyslog.d/ directory Sample 60-nginx.conf file #nginx.* @@127.0.0.1:514 :syslogtag, isequal, "[nginx]" /var/log/nginx-syslog/nginx-access-log.log References: F5 Distributed Cloud Global log receiver supports many log receivers natively: F5 Distributed Cloud Technical Knowledge page on "Configure Global Log receiver" Prerequisites: An external log collection system reachable publicly. The following IP address ranges are required to be added to your firewall's allowlist: 193.16.236.64/29 185.160.8.152/29Load Balancing TCP TLS Encrypted Syslog Messages
Syslog messages sent via TCP are not always evenly distributed among backend syslog servers because multiple syslog messages can be sent in a single TCP connection. This article utilizes the F5 BIG-IP Generic Message Routing Framework (MRF) to evenly distribute syslog messages among backend syslog pool members. This solution also uses TLS to protect the confidentiality of the syslog messages. This article is based off the work done by Mark Lloyd in this DevCentral Technical Article: A Simple One-way Generic MRF Implementation to load balance syslog message. In his article, Mark explains how to setup Generic Message Routing Framework (MRF) to distribute syslog messages sent via TCP to a pool of syslog servers. This article adds the necessary configuration to TLS encrypt, decrypt, and re-encrypt the messages. This was tested on BIG-IP version 17.1.0.1. The first step is to define the message routing protocol. The difference between the default protocol (genericmsg) is the field no-response must be configured to yes if this is a one-way stream. Otherwise, the server side will allocate buffers for return traffic that will cause severe free memory depletion. Note: The message-terminator is set to "%0a", this represents a newline character in hex and is used to separate the syslog messages. This value can be changed if a different delineator is required. ltm message-routing generic protocol simple_syslog_protocol { app-service none defaults-from genericmsg description none disable-parser no max-egress-buffer 32768 max-message-size 32768 message-terminator %0a no-response yes } An iRule must be configured on both the Virtual Server and Generic Transport Config. This iRule must be linked as a profile in both the virtual server and the generic transport configuration. ltm rule mrf_simple { when CLIENT_ACCEPTED { GENERICMESSAGE::peer name "[IP::local_addr]:[TCP::local_port]_[IP::remote_addr]:[TCP::remote_port]" } when SERVER_CONNECTED { GENERICMESSAGE::peer name "[IP::local_addr]:[TCP::local_port]_[IP::remote_addr]:[TCP::remote_port]" } } The next item to configure is the generic transport config. The generic transport config has the generic protocol configured along with the iRule to setup the server-side peers. A server-side SSL profile is also configured here to TLS encrypt the traffic to the backend syslog servers. ltm message-routing generic transport-config simple_syslog_tcp_tc { ip-protocol tcp profiles { serverssl-insecure-compatible { } simple_syslog_protocol { } tcp { } } rules { mrf_simple } } Nodes are defined for the backend Syslog servers. ltm node 10.1.20.201 { address 10.1.20.201 } ltm node 10.1.20.202 { address 10.1.20.202 } A pool is created containing the nodes. ltm pool syslog_pool { members { 10.1.20.201:514 { address 10.1.20.201 } 10.1.20.202:514 { address 10.1.20.202 } } } The next step is to create the generic message routing peer. This peer is used to identify the pool of syslog servers that the syslog messages will be routed to. ltm message-routing generic peer simple_syslog_peer { pool syslog_pool transport-config simple_syslog_tcp_tc } Now that the peer is defined, a generic route can be created to send traffic to the peer. ltm message-routing generic route simple_syslog_route { peers { simple_syslog_peer } } A generic router is configured with the generic route. ltm message-routing generic router simple_syslog_router { app-service none defaults-from messagerouter description none ignore-client-port no max-pending-bytes 23768 max-pending-messages 64 mirror disabled mirrored-message-sweeper-interval 1000 routes { simple_syslog_route } traffic-group traffic-group-1 use-local-connection yes } A client-ssl profile is configured to associate the certificates to the Virtual Server. ltm profile client-ssl syslog-ng_client { app-service none cert-key-chain { syslog-ng-new_f5demos_ca_0 { cert syslog-ng-new chain f5demos_ca key syslog-ng-new } } defaults-from clientssl inherit-ca-certkeychain true inherit-certkeychain false } A virtual server is created to receive incoming TLS-encrypted TCP syslog messages. ltm virtual mrftest_simple { creation-time 2024-10-08:09:37:52 destination 10.1.10.101:514 ip-protocol tcp last-modified-time 2024-10-08:13:55:49 mask 255.255.255.255 profiles { simple_syslog_protocol { } simple_syslog_router { } syslog-ng_client { context clientside } tcp { } } rules { mrf_simple } serverssl-use-sni disabled source 0.0.0.0/0 source-address-translation { type automap } translate-address enabled translate-port enabled vs-index 2 } Conclusion This example is from a use case where a single syslog client was sending to a TCP load balancer. The load balancer was not evenly distributing the load among the backend servers because multiple messages were being sent as part of a single TCP connection. This solution utilizes a generic Message Routing Framework to evenly distribute TCP syslog messages. TLS encryption is also used on the client to load balancer connection as well as the load balancer to backend server connection to protect the confidentiality of the syslog messages.SSL Orchestrator Enhanced Uses Case: Remote Logging
Introduction This use case allows you to configure the BIG-IP SSL Orchestrator to send detailed logging to a remote Syslog server. Logging is an important aspect of SSL Orchestrator operation and troubleshooting. The volume of data created by debug logging is significant and should ideally be sent off-box for analysis and archiving. The following instructions demonstrate how to configure Remote Logging. Logging Level Logging verbosity is configured in the BIG-IP Configuration Utility. Under SSL Orchestrator select Configuration > Logs > Settings. Logging verbosity is set to Error by default. Change this to Debug for the Per-Request Policy and SSL Orchestrator Generic. Click Save when done. Note: For simplified logging that combines each connection flow into a single summary log, only enable SSL Orchestrator Generic at level Informational of higher. These log settings are Global and can be over ridden by per-Topology logging settings. Create a Pool for the Syslog server Under Local Traffic select Pools. Click Create. Give it a name, Remote_syslog_pool in this example. Give the Node a Name, syslog_server in this example. Enter the IP address of the syslog server and port 514 for the Service Port. Click Add. Note: 514 is the common port for syslogd but may be different in your environment. If desired, add a Health Monitor like gateway_icmp. Use the << to move it from Available to Active. Click Finished when done. Create Logging Destination Under SSL Orchestrator select Configuration > Logs > System. Then select Configuration > Log Destinations. Click Create. Give it a name, remote_syslog in this example. Select Remote High-Speed Log as the Type. Note: The Remote High-Speed Log (HSL) uses the data plane while Remote Syslog uses the management plane. HSL logging is preferred due to better, sustained performance. For more information on HSL click here. For Pool Name select the Pool created previously, Remote_syslog_pool in this example. Set the Protocol to UDP or TCP (typically UDP). Click Finished. Configure the Log Publisher From the same screen click Configuration > Log Publishers. Click on sys-sslo-publisher to edit it. Select the local-syslog and click the >> to move it to Available. Select the remote_syslog and click << to move it to Selected. Click Update when done. The configuration is now complete. Detailed logs should now be sent to your Syslog server. Verify it’s Working Check the Syslog Pool Statistics. From the BIG-IP Configuration Utility select Local Traffic > Pools. Select Statistics. If it is working you should see a non-zero value for Bits and Packets. Check your Syslog Server to verify it is receiving logs from SSL Orchestrator. In this example I’m running a packet capture on the Syslog Server to check that packets are being sent from the BIG-IP to the Syslog Server. In the example above you can see that the BIG-IP (10.0.0.1) is sending packets to the Syslog Server (10.0.0.2) on UDP port 514. You can also see the details of the Syslog message in the circle. Note: BIG-IP SSL Orchestrator needs a Self IP Address in order to send detailed logging to the Syslog Server. If deployed in Layer 2 mode you will need to configure a new Self IP Address. You cannot assign an IP address to an interface in an L2 vwire group. If deployed in Layer 3 mode you can use an existing Self IP Address as long as it can reach the Syslog Server. Ideally though, the Syslog traffic should not be on the same interface(s) as client/server traffic. In this example BIG-IP is configured with the Self IP 10.0.0.1 which is on the same subnet as the Syslog Server at IP address 10.0.0.2. Summary In this SSL Orchestrator Use Case you learned how to enable detailed logging on BIG-IP and have the logs sent to a remote Syslog Server.Changing the BIG-IP Default Syslog-NG Facilities
DevCentral community member geffr had a problem. The BIG-IP Application Security Manager module logs to the local3 facility but he needs to send them to the local7 facility on a remote server. Before giving up entirely, he posted to this thread in the Monitoring & Management group forum, where user nitass helped him jump through the syslog-ng hoops (click here for tips & tricks on syslog-ng) to the working solution posted below. It’s pretty straight forward. Define a template, a filter, and a destination, and then put the pieces together in a log statement.
20 Lines or Less #22
What could you do with your code in 20 Lines or Less? That's the question I ask (almost) every week, and every week I go looking to find cool new examples that show just how flexible and powerful iRules can be without getting in over your head. Wow! It's been a while since I was able to pull the cycles away from other things to get to the 20LoL. Many apologies to any of you out there that have missed it. Things here have been amazingly busy (in a good way) and I just haven't made the time. There are so many cool iRules to cover, I'll have plenty of fodder in weeks to come, so hopefully things will be more regular. This week we'll cover three cool, short iRules, as always, but this week we have a special request for the 20LoL that I'll be including just for DevCentral's own Joe Pruitt. IE8 Compatibility http://devcentral.f5.com/s/Default.aspx?tabid=53&forumid=5&tpage=1&view=topic&postid=56407#56534 Here's a cool iRule that Joe requested get tossed into the mix for the 20LoL, and I'm more than happy to oblige. This cool rule inspects an incoming user's request to see if they're using IE8. If so, it inserts the necessary headers to force them into IE7 compatibility mode. This can be handy for now since not everything works perfectly with IE8 just yet. when HTTP_REQUEST { # Check if User-Agent header is IE 8 if { [HTTP::header User-Agent] contains "MSIE 8" } { set IE8Resp 1 } else { set IE8Resp 0 } } when HTTP_RESPONSE { if { $IE8Resp } { log "[IP::client_addr]:[TCP::client_port] inserting header" HTTP::header insert "X-UA-Compatible" "IE=EmulateIE7" } } Redirecting from Java Strings http://devcentral.f5.com/s/Default.aspx?tabid=53&forumid=5&postid=56263&view=topic This request in the forums was to expand the scope of an originally straight-forward HTTP redirect rule to do a little bit more inspection, and route people to different places based on the results. It's great to see people not only using but modifying and expanding their iRules as time goes on. Good stuff. when HTTP_REQUEST { if { [HTTP::path] equals "/" }{ if { not ([string tolower [HTTP::query]] starts_with "user=soviet") }{ if { [HTTP::header "User-Agent"] contains "soviet" }{ HTTP::redirect "http://soviet.com/" } } } } Display log messages to your browser http://devcentral.f5.com/s/wiki/default.aspx/iRules/webLog_-_Display_log_messages_to_Web_Browser.html The original version of this iRule comes straight from the iRules CodeShare in the Wiki, linked to above. It's a cool idea of being able to display the messages your request is generating from the server directly to a browser, rather than forcing a developer or admin to repeatedly view the log file for updates while testing / coding. This would be used with some controls in place and in moderation of course, but still a very cool idea. It took a little tinkering to get it under 20 lines, but the functionality should be the same, and we all know I love "squeezing" code. ;) when HTTP_REQUEST { append ::webLogMsg "HTTP URI: [HTTP::uri]" } when HTTP_RESPONSE { if {([IP::client_addr] equals "x.x.x.x") && ([info exists ::webLogMsg]) } { if {[HTTP::header "Content-Length"] > 2000} { HTTP::collect 2000 } else { HTTP::collect [HTTP::header "Content-Length"] } } } when HTTP_RESPONSE_DATA { regexp -indices {(?i) ]*>} [HTTP::payload] BodyLocation HTTP::payload replace [lindex $BodyLocation 1] 1 [concat {> } \ "Client Port: [TCP::client_port] $::webLogMsg" { }] unset ::webLogMsg HTTP::release } Thanks for reading the 20LoL this week. Hopefully you'll be back for more awesome iRules examples in less than 21 lines. #Colin Listening to: Tool - ÆNIMA - Forty Six and 220 Lines or Less #9
What could you do with your code in 20 Lines or Less? That's the question I ask every week, and every week I go looking to find cool new examples that show just how flexible and powerful iRules can be without getting in over your head. This week I've got a combination of entries from our awesome forum users, and a rule I wrote a while back to meet a certain need at the time. We're almost at 10 editions of the 20LoL, and I'm looking forward to many more. Hopefully you're still finding it interesting and useful. Shoot me a line and let me know what's good, what's bad, what can be better and what you want to hear about. In the meantime, here's this week's 20 Lines or Less Multi-Conditional Redirect http://devcentral.f5.com/s/Default.aspx?tabid=53&forumid=5&postid=25219&view=topic Hoolio delivers this short and sweet iRules in the forums to show how you can use multiple pieces of data to decide when to perform a redirect. Not only does he make use of a normal string comparison, but also an IP::addr comparison against the client's IP address. So in one line you're getting two comparisons on two different pieces of data. This is a good example for someone looking to redirect only a small subset of people, based on multiple pieces of data. when HTTP_REQUEST { if { [string tolower [HTTP::path]] ends_with "/_grid/print/print_data.aspx" \ and (not ([IP::addr [IP::client_addr]/8 equals 10.0.0.0]))} { HTTP::redirect "http://google.com" } } Syslog Priority Rewriting This is a variation on some actual code I wrote a while back to translate the syslog priority numbers when needed. Depending on the different syslog configurations, these numbers may not line up. This can be a problem when you're trying to aggregate many syslog systems into one main log server. This iRule shows how you can catch these messages inline and modify them with whatever equation fits your environment. when CLIENT_DATA { set pri [regexp -inline {} [UDP::payload] ] set newPri [expr ( ($pri - (($pri / 6) * 8) ) ) ] regsub $pri [UDP::payload] $newPri newPayload UDP::payload replace 0 [UDP::payload length] $newPayload } Duplicate Cookie Definitions http://devcentral.f5.com/s/Default.aspx?tabid=53&forumid=5&postid=25215&view=topic Going back to the forums, it seems that hoolio is at it again. In this cool example he shows a fellow community member how to check for and remove multiple Set-Cookie entries with the same name. This way they can ensure that there is only one cookie present, regardless of how many times different apps may have tried to set it. This one looks a little long, but remove the comments and some of the white space, and it's under 20 lines...I checked. when HTTP_RESPONSE { # Insert some test response headers HTTP::header insert Set-Cookie {SESSIONID=AAAAAAAA; domain=.domain.com; path=/path/1} HTTP::header insert Set-Cookie {keeper=don't delete; domain=.domain.com; path=/path/2} HTTP::header insert Set-Cookie {SESSIONID=BBBBBBBB; domain=.domain.com; path=/path/3} HTTP::header insert Set-Cookie {SESSIONID=CCCCCCCC; domain=.domain.com; path=/path/4} log local0. "Set-Cookie header values: [HTTP::header values Set-Cookie]" log local0. "First Set-Cookie header which starts with SESSIONID: \ [lsearch -glob -inline [HTTP::header values Set-Cookie] "SESSIONID*"]" log local0. "Last Set-Cookie header which starts with SESSIONID: \ [lsearch -glob -inline -start end [HTTP::header values Set-Cookie] "SESSIONID*"]" set set_cookie_header [lsearch -glob -inline -start end [HTTP::header values Set-Cookie] "SESSIONID*"] log local0. "\$set_cookie_header: $set_cookie_header" # Remove all SESSIONID cookies while {[HTTP::cookie exists SESSIONID]}{ HTTP::cookie remove SESSIONID } log local0. "Set-Cookie values: [HTTP::header values Set-Cookie]" # Re-insert the last SESSIONID Set-Cookie header HTTP::header insert Set-Cookie $set_cookie_header log local0. "SESSIONID cookie: [HTTP::cookie SESSIONID]" } There you have it, 3 more examples in under 60 lines of code. Keep checking back every week to see what cool things can be done in just a few keystrokes. Many thanks to the awesome community and the people posting these examples. You're truly making DC a great place to be. #Colin
