Forum Discussion
smp_86112
Cirrostratus
CirrostratusPersistent Connection Forwarded to Wrong Pool Member
v9.3.1HF8
(I haven't really studied this behavior closely before, so please go light on me.)
It seems we have identified an issue where HTTP clients with a BigIP Cookie are being forwarded to the wrong pool member. Upon review of the logs, we see the same BigIP cookie header value on both pool members. The clients are behind a proxy server, so we suspect that multiple client sessions are being pipelined into a single TCP session between the LTM and the proxy. The VIP is pretty vanilla, and so is the persistence profile which I have included below for reference.
How do you troubleshoot something like this? The GUI doesn't show the persistence records, so I can't confirm from that perspective. Is there a different type of persistence I could use that would prevent this problem from happening at all? What other mitigation strategies are out there?
virtual app {
destination :80
ip protocol tcp
profile default_http tcp
persist cookie
pool app
}
profile persist cookie {
mode cookie
cookie mode insert
cookie expiration 0d
cookie hash offset 0
cookie hash length 0
cookie name none
rule none
mirror disable
timeout 0
}
pool app {
lb method member predictive
min active members 1
monitor all dtac.deere.com
member :
member :}
HamishCirrocumulus
For cookie persistence, there is no information stored on the F5 itself for the persistence (At least none that I've found, and F5 tell me this as well). The information for persistence is stored solely within the cookie. Because it's un-encrypted, you can decode the cookie and see what the value says.
If the poolmember listed in the cookie isn't valid (i.e. isn't a valid and up poolmember), then a new poolmember will be chosen from the list of available poolmembers.
How much traffic are we talking about here? If there too much to do a tcpdump out to a file (i.e.
tcpdump -i 0.0 -n -p -s 0 -w /var/tmp/somefile.dump host and host and port 80
)
to view the traffic both between the clients & VS and between the F5 and the poolmembers?
Then you can verify whether things are working as designed or not. (i.e. use wireshark or similar to look at the connections. Is the cookie being set by the F5 the same as the cookie being presented back to the F5 by the client? Is the proxy inserting an XForwarded-For header? Does that IP change in the header on the same connection?
H
H
hoolioThose are good troubleshooting steps, but I think this is an expected problem. The issue occurs when multiple clients access the VIP over the same TCP connection with different persistence records. You can add a OneConnect profile to fix this:
http://devcentral.f5.com/wiki/default.aspx/AdvDesignConfig/oneconnect.html
OneConnect with HTTP
Without OneConnect enabled, persistence data is examined only in the first request of a Keep-Alive connection, so if multiple requests are sent on the same clientside Keep-Alive connection, LTM will persist them all to the same destination as the first unless a OneConnect profile is applied (even if logic contained in an iRule dictates otherwise).
If the connections from LTM to the pool use SNAT, you can use a /0 source mask on the OC profile. Else, you should use a /32 mask to ensure the server logs show the correct client IP address.
Aaron
smp_86112I think we have verified the behavior with tcpdump and a review of the web server logs - what we were lacking is an explanation of what we were seeing. Ultimately I found SOL7964 which seems to be describing the symptoms, and also lays out the two potential F5 options we have. Thanks for the feedback.