How "reliable" is the HSL logging profile capability? Is it known that there may be requests that do not get logged, because of the F5 not sending the UDP packet? Or, if there are requests not being logged, is basically definite that the UDP packets are being dropped, either at the network level or by the log collector (Kiwi syslog server, if it matters)?
We have a specific data flow that involves two layers of F5 load balancing:
Client -> F5-1 -> F5-2 -> real server
We have an HSL request logging profile in place for the VIPs on both F5s. With perfect logging, we would expect to see a 1:1 ratio of requests hitting F5-1 and F5-2 - but we don't, we see many logged requests hitting F5-2 without a corresponding request logged on F5-1. We know for a fact that those requests did come through F5-1, because on F5-2 we log both the client-ip and X-Forwarded-For header, and can see F5-1 the client-IP, and both the original client IP and the F5-1 in the XFF header.
In trying to identify the root cause of the missing log messages, are there any logs on the F5s that would indicate if logging is failing, or a rate has been exceeded, or any other kind of issue?
We do only have 1 log collector in the associated pool - but the request rate is pretty low, only about 10-15 requests/second; and if that server were being overwhelmed, we would expect to see even drops, not mostly/all F5-1's messages being dropped.
In terms of trying to improve reliability, would switching to TCP be advised?
Any other suggestions, for either locating the issue, or improving reliability?