High speed logging and round robin load balancing issues

Hi,

We have a couple of F5 servers that have an iRule that uses HSL to send log messages to a pool of remote rsyslog daemons.

The rsyslog daemons are added to one pool, and the pool is configured to distribute log messages in a round robin way.

All the nodes in the pool are up and running and are in a green state.

I would expect that the log messages get distributed evenly accross all nodes in that logging pool.

However it looks like the round robin balancing is not used.

All the messages sent with HSL are always sent to the same rsyslog daemon.

Only when logrotation kills rsyslog for a short term another rsyslog daemon is selected.

Is there some other stickyness setting that we're forgetting? Or is HSL not supporing the load balancing configuration of the pool?

Or is there another setting that I'm forgetting?

Any ideas?

Thanks,

Pieter

management

monitoring

17 Replies

hoolio
Cirrostratus
Feb 07, 2013
Hi Pieter,

Can you post the HSL pool config using 'tmsh list ltm pool hsl_pool' and reply with the anonymized output?

Thanks, Aaron
Pieter_72260
Nimbostratus
Feb 07, 2013
Hi Aron,

Below is the pool config:

ltm pool hsl_pool {

members {

10.197.15.45:514 {

session monitor-enabled

}

10.197.15.46:514 {

session monitor-enabled

}

}

monitor tcp

}

Once colleague thinks it might have something to do with the persistency settings that are used in the irule:

when HTTP_REQUEST priority 100 {

if { "POST" eq [HTTP::method] } {

persist none

} else {

persist carp [HTTP::uri]

}

}

Thanks in advance,

Pieter
What_Lies_Bene1
Cirrostratus
Feb 08, 2013
The persistence method you mention is for HTTP traffic using the VS, not the HSL traffic. The HSL Pool configuration looks good. Do you know if the F5 is opening multiple connections or just one?
nitass
Employee
Feb 10, 2013
what event is HSL::open in? is it CLIENT_ACCEPTED?

if so, have you ever tried to move HSL::open to HTTP_REQUEST (i just want to check if what i guess is correct)?
Pieter_72260
Nimbostratus
Feb 12, 2013
Hi nitass,Steve,

The HSL::open is indeed in CLIENT_ACCEPTED and we use HSL::send in CLIENT_CLOSED,HTTP_RESPONSE and LB_FAILED.

I'll check with our SysEng team if they can see connections, or health checks being executed.

Thanks,

Pieter
Pieter_72260
Nimbostratus
Feb 12, 2013
And some more info.

We've done a TCP dump on the syslog daemons (using tcpdump -w f5.cap -s 0 port 514)

And for both the rsyslog daemons we can see the TCP health checks from both F5 loadbalancers (we have 2 for failover purposes)

So all that seems right, but still we only get logs on one rsyslog daemon.

Thanks,

Pieter
nitass
Employee
Feb 12, 2013
i did test on my 10.2.4 unit and got the same issue (i.e. hsl log is sent to only one pool member. additionally, only 3-way handshake is done). i think it could be a bug.

have you opened a support case? if not yet, would you mind logging a case to verify?
Pieter_72260
Nimbostratus
Feb 12, 2013
Hi Nitass,

We've created a support case: C1289376

I've seen this behavior on some of our load balancer pair's.

But not all of them seem to have an issue with it.

And once we restart the rsyslog daemons for these LB pairs the issue happens as well.

Thanks,

pieter
nitass
Employee
Feb 12, 2013
But not all of them seem to have an issue with it. are they using tcp syslog? i have not yet tested but understanding udp syslog could be okay.
Pieter_72260
Nimbostratus
Feb 12, 2013
They are all using TCP syslog. I'll see if can try and test UDP syslog.