Forum Discussion
Chris_Phillips
Nimbostratus
Nimbostratusmonitor timeouts vs actual behaviour
Hi guys,
the default http health monitor (v10.2.4) polls on a 5 second interval, timeout of 16 seconds. To me, this says that every 5 seconds a monitor will fire, should no monitor be successful for 16 seconds then the pool mmeber is down.
Yet this really REALLY doesn't match what happens on the network to a huge extent:
pool blah_pool {
monitor all http
members 1.2.3.4:1234 {}
}
`
a tcpdump shows:
`11:01:36.761159 IP 10.101.131.4.35514 > 1.2.3.4.1234: S
11:03:13.742647 IP 10.101.131.4.46160 > 1.2.3.4.1234: S
11:03:16.742445 IP 10.101.131.4.46160 > 1.2.3.4.1234: S
11:03:22.742838 IP 10.101.131.4.46160 > 1.2.3.4.1234: S
11:03:34.741285 IP 10.101.131.4.46160 > 1.2.3.4.1234: S
11:03:58.740435 IP 10.101.131.4.46160 > 1.2.3.4.1234: S
11:04:46.736725 IP 10.101.131.4.46160 > 1.2.3.4.1234: S
11:06:23.738147 IP 10.101.131.4.48428 > 1.2.3.4.1234: S
11:06:26.737763 IP 10.101.131.4.48428 > 1.2.3.4.1234: S
11:06:32.737102 IP 10.101.131.4.48428 > 1.2.3.4.1234: S
11:06:44.735753 IP 10.101.131.4.48428 > 1.2.3.4.1234: S
so we have only one single TCP attempt at one time, so not every 5 seconds, and whilst the monitor will mark a node down after 16 seconds still, the tcp connection is still going to try to continue until the tcp/ip stack times it out. So once it's down after 16 seconds it's still got a huge wait before it tries again, no new connection will try to connect until the single current one finishes. so if, for some (presumably pretty stupid) reason the specific connection is not being replied to, maybe a weird FW rule or IPS action) LTM won't be able to check status on a new connection for three minutes and 10 seconds.
I've also seen equivalent behaviour with an http GET just not being replied to, again having to wait until the TCP connection is reset, or the webserver finally responds well, well after the "timeout" period has expired before the monitor will fire again. Testing just now, I see the HTTP monitor just crudely stuffing additional GET's down the same connection that's still waiting for a response, what's that all about??
I can't make any sense of this, and, TBH, has gone right against all the things I've designed for, sticking to the 3n+1 rule etc. What merit does 3n+1 have in this sort of situation? I see no logic in it at all if additional monitors can't run in parallel. Who would want to be forwarding to a web server that is routinely taking, say, 15 seconds to reply (3n+1 - 1s) when all the other members in a pool take 0.01s to serve the same gif file? Shouldn't a timeout actually always be something like 4 seconds (to at least give time for 2 SYN's to hit the back end? Even in that case though, I'm still stuffed until the next connection is allowed to be attempted.
Any thoughts on this would be appreciated!
natheCirrocumulus
Chris, I've just checked my 10.2.4 big-ip and did a tcpdump on a pool member with a http monitor (as you've got in the example above). The connections are exactly 5secs apart on mine and I was able to mark the pool member down after 16secs and then back up again on a single poll (less than 5secs). Unless I've missed anything else to recreate your testing then it seems to be working as it should. Sorry I haven't got an answer for you but wanted to reassure you that it should work as documented. Have you got any other traffic muddying the water? N
IheartF5_45022Nacreous
3n+1 is the recommendation from F5 only. It's based on their perception of a 'safe' number of tries before marking a pool member down. If you find that a different interval/timeout works better for you in your environment - go for it and create a new monitor based on the default one.
Having said that I have seen and also wondered at the behaviour of monitors when the servers are not behaving optimally (as far as I can see they do behave exactly as per the doco when they complete handshakes and other responses efficiently).
The F5 is supposed to RST a TCP connection after 3 retransmissions, however we can see 6 SYN transmissions for a single client port in your tcpdump above. Maybe they don't follow the tcp profile settings?
In addition I totally agree with your observation "Testing just now, I see the HTTP monitor just crudely stuffing additional GET's down the same connection that's still waiting for a response, what's that all about??" - what indeed? It's disturbed me for a long time and I've been meaning to post to this forum about it. It's like it's doing HTTP request pipelining, and I've only just realised that in fact pipelining is enabled by default in http profile, so I wonder if disabling pipelining in the profile will prevent the request stuffing behaviour of the monitor? I will try tomorrow. The problem with this behaviour is that if the single TCP connection that gets opened is a not-very-well one, we really only get one try at a response - I mean if it hasn't responded to the first request it's hardly likely to respond to the 2nd.
Another thing I've thought of to try is to apply 3 http-based monitors with interval/timeout 5/6 to the same pool, and say that a minimum of 2 have to pass in order to mark the member up. This would fix the request stuffing and perhaps the single tcp connection with 5 retransmits. If anyone else has experimented with this kind of thing please let us know!!
Chris_PhillipsThe 3n+1 makes sense to *some* extent if things work as documented, but they don't. I understood it was to have three failed checks in a row, which yeah sure, a consistent failure is proven. But with only a single connection possible, the logic doesn't even begin to exist, let alone be sub-optimal. Health monitors won't follow a TCP or HTTP profile at all. Profiles are against the VIP, so pool member availability wouldn't be relevant to it. It's just bog standard OS TCP/IP stack defaults I understand, so I could presumably go tune the actual kernel parameters but somehow I don't think so! As far as monitoring a pool of 50 web apps in the rack next door, with a 10GE interconnect, if I don't get a TCP connection within 0.01 seconds, I'm realistically not going to get one, and if I did, the vary fact I had to retry should be enough to not want to. Yet that is totally blocked off what what I can do. The way things are documented and taught really seem massively different to the reality.
- Chris_Phillipswhat you're describing seems to be consistent. When things ARE happy, yes it'll poll every 5 seconds, and the member will go down after 16 if configured to do so. But you can't then bring it back up until you can poll it again, which requires the existing failing connection to stop. What things look like under the hood doesn't reflect the outward presentation of the member availability. If you rig things so that the TCP monitor goes unanswered, not reset, what do you see then? I presume you'll see the same huge huge delay. This kit has been upgraded over the years from something like 9.0.5, maybe there's some junk settings hidden away somewhere that could be changing things? I'd certainly be interested to hear more about the HTTP side of things, as that seems much more blatant a behavior to see.