Performance optimalization of message based load balancing.

There are several benefits to message based load balancing (mblb) rather than the traditional connection based load balancing, as described in Load-Balancing Syslog Messages Part 1. This implementation works well, however in a high volume logging environment it may be beneficial to consider multi-message based load balancing (mmblb) to enhance performance.

At the core of the message-based load balancing there is an iRule which carves out new-line separated messages from the TCP stream and submits them to a load balancing decision.

The "original" iRule:

when CLIENT_ACCEPTED {
   TCP::collect
}
when CLIENT_DATA {
   set minimum_message_length 1
   while { [TCP::payload] contains "\x0a" && [TCP::payload length] > $minimum_message_length } {
      set m [getfield [TCP::payload] "\x0a" 1]
      set length [expr [string length $m] + 1]
      TCP::release $length
      TCP::notify request
   }
  TCP::collect
}

This iRule will carve out log messages one by one and for each log message perform a load balancing decision. With respect to performance this does not scale well.

By modifying the "original" iRule, a multi-message based load balancing (mmblb) iRule has been developed. The main changes are:
1) Expanding the TCP collect frame size to fit more messages for processing.
2) Instead of carving message by message, finding the last new-line character and load balance all messages up to this point.

The "modified" iRule (mmblb) using collection size of 150KB (configurable):

when CLIENT_ACCEPTED {
   TCP::collect 150000
}
 
when CLIENT_DATA {
   set mess [TCP::payload]
   # string last function will return -1 if no new-line character found in $mess  
   set messlength [expr [string last "\x0a" $mess] + 1]
   if { $messlength > 0 } {
      TCP::release $messlength
      TCP::notify request
   }
  TCP::collect 150000
}

This iRule (mmblb) allows for multiple log messages, determined by the TCP collection size, to be carved out in one chunk and then one load balancing decision before collecting more data from the TCP stream.

Note that the pool member data volume statistics will be more uneven for smaller data volumes (KB/MB) until larger volume sizes are displayed (GB+).

For a specific set of test messages, the modified iRule (mmblb) significantly reduced the number of Executions and number of Instruction Cycles according to internal resource statistics.

This iRule is successfully deployed in a 5B+ log messages per day environment using a F5 cluster and 5 virtual Rsyslog backend servers.