Two virtual servers go down after an upgrade
I'm wondering if anyone has seen this behaviour before. After I perform an upgrade, two virtual servers out of about 10 go down. The applications using these VSs stop working, from the looks of it outbound traffic out to the internet stops.
The health monitor used is the default HTTPS. I've created a custom monitor to GET a file from the backend pool members which marks the nodes as up, but the apps still don't work. The backend servers are running IIS10.0.
Some versions of the F5 software work, most do not. Working versions are 18.104.22.168, 14.1.4, and 22.214.171.124. All other versions seem to not work. As soon as the upgraded device is made active (in the HA pair) the VSs go down. Packet captures don't seem to show the issue, but they do indicate for some reason there's a 75 second+ pause in the response from the pool members. This isn't there when one of the working versions is active so I don't think this is an issue with the pool member.
The traffic passes through two sets of Checkpoint firewalls, and is NATed each time on these firewalls on the way to the internet.
Could anyone provide information as to why this would work with some versions of the BIG-IP software, and not others?
This looks to have possibly been this bug: https://my.f5.com/manage/s/article/K85805058
The actual issue was on pool members behind the F5 - they used the F5s as a gateway to get to the internet using an IP forwarding VS. When the issue occurred these pool members were unable to get to the internet. It looks like the standard HTTPS health checks failed because the pool members were timing out trying to load internet content.
After further examination of packet captures it was observed there was possibly async traffic (based on MACs observed).
The fix was to create a new FastL4 profile and make sure 'loose init' and 'loose close' were enabled. This profile was then used on the ip forwarding VS, and it looks like this has solved the issue.
126.96.36.199 was installed and is so far working fine.