Load Balancing TCP TLS Encrypted Syslog Messages

Syslog messages sent via TCP are not always evenly distributed among backend syslog servers because multiple syslog messages can be sent in a single TCP connection. This article utilizes the F5 BIG-IP Generic Message Routing Framework (MRF) to evenly distribute syslog messages among backend syslog pool members. This solution also uses TLS to protect the confidentiality of the syslog messages.

This article is based off the work done by Mark Lloyd in this DevCentral Technical Article: A Simple One-way Generic MRF Implementation to load balance syslog message. In his article, Mark explains how to setup Generic Message Routing Framework (MRF) to distribute syslog messages sent via TCP to a pool of syslog servers. This article adds the necessary configuration to TLS encrypt, decrypt, and re-encrypt the messages.

This was tested on BIG-IP version 17.1.0.1.

The first step is to define the message routing protocol. The difference between the default protocol (genericmsg) is the field no-response must be configured to yes if this is a one-way stream. Otherwise, the server side will allocate buffers for return traffic that will cause severe free memory depletion. Note: The message-terminator is set to "%0a", this represents a newline character in hex and is used to separate the syslog messages. This value can be changed if a different delineator is required.

ltm message-routing generic protocol simple_syslog_protocol {
    app-service none
    defaults-from genericmsg
    description none
    disable-parser no
    max-egress-buffer 32768
    max-message-size 32768
    message-terminator %0a
    no-response yes
}

An iRule must be configured on both the Virtual Server and Generic Transport Config. This iRule must be linked as a profile in both the virtual server and the generic transport configuration.

ltm rule mrf_simple {
when CLIENT_ACCEPTED {
        GENERICMESSAGE::peer name "[IP::local_addr]:[TCP::local_port]_[IP::remote_addr]:[TCP::remote_port]"
    }
 
when SERVER_CONNECTED {
        GENERICMESSAGE::peer name "[IP::local_addr]:[TCP::local_port]_[IP::remote_addr]:[TCP::remote_port]"
        
    }
}

The next item to configure is the generic transport config. The generic transport config has the generic protocol configured along with the iRule to setup the server-side peers. A server-side SSL profile is also configured here to TLS encrypt the traffic to the backend syslog servers.

ltm message-routing generic transport-config simple_syslog_tcp_tc {
    ip-protocol tcp
    profiles {
        serverssl-insecure-compatible { }
        simple_syslog_protocol { }
        tcp { }
    }
    rules {
        mrf_simple
    }
}

Nodes are defined for the backend Syslog servers.

ltm node 10.1.20.201 {
    address 10.1.20.201
}
ltm node 10.1.20.202 {
    address 10.1.20.202
}

A pool is created containing the nodes.

ltm pool syslog_pool {
    members {
        10.1.20.201:514 {
            address 10.1.20.201
        }
        10.1.20.202:514 {
            address 10.1.20.202
        }
    }
}

The next step is to create the generic message routing peer. This peer is used to identify the pool of syslog servers that the syslog messages will be routed to.

ltm message-routing generic peer simple_syslog_peer {
    pool syslog_pool
    transport-config simple_syslog_tcp_tc
}

Now that the peer is defined, a generic route can be created to send traffic to the peer.

ltm message-routing generic route simple_syslog_route {
    peers {
        simple_syslog_peer
    }
}

A generic router is configured with the generic route.

ltm message-routing generic router simple_syslog_router {
    app-service none
    defaults-from messagerouter
    description none
    ignore-client-port no
    max-pending-bytes 23768
    max-pending-messages 64
    mirror disabled
    mirrored-message-sweeper-interval 1000
    routes {
        simple_syslog_route
    }
    traffic-group traffic-group-1
    use-local-connection yes
}

A client-ssl profile is configured to associate the certificates to the Virtual Server.

ltm profile client-ssl syslog-ng_client {
    app-service none
    cert-key-chain {
        syslog-ng-new_f5demos_ca_0 {
            cert syslog-ng-new
            chain f5demos_ca
            key syslog-ng-new
        }
    }
    defaults-from clientssl
    inherit-ca-certkeychain true
    inherit-certkeychain false
}

A virtual server is created to receive incoming TLS-encrypted TCP syslog messages.

ltm virtual mrftest_simple {
    creation-time 2024-10-08:09:37:52
    destination 10.1.10.101:514
    ip-protocol tcp
    last-modified-time 2024-10-08:13:55:49
    mask 255.255.255.255
    profiles {
        simple_syslog_protocol { }
        simple_syslog_router { }
        syslog-ng_client {
            context clientside
        }
        tcp { }
    }
    rules {
        mrf_simple
    }
    serverssl-use-sni disabled
    source 0.0.0.0/0
    source-address-translation {
        type automap
    }
    translate-address enabled
    translate-port enabled
    vs-index 2
}

Conclusion

This example is from a use case where a single syslog client was sending to a TCP load balancer. The load balancer was not evenly distributing the load among the backend servers because multiple messages were being sent as part of a single TCP connection. This solution utilizes a generic Message Routing Framework to evenly distribute TCP syslog messages. TLS encryption is also used on the client to load balancer connection as well as the load balancer to backend server connection to protect the confidentiality of the syslog messages.