AFter upgrade from 11.4.1 HF4 to 11.5.1 HF5 Getting RST after SSL connection setup, cert and keys are passed

I presume you would have raised a support case by now? Someone will have to look at the network captures to tell what TLS protocol is being used, and if TLS is the issue, then what action can be taken. For monitors, the device should be able to downgrade the connection to a lower tls version if the server sends a proper ssl alert indicating an unsupported protocol version. If the server merely resets the connection, it could simply be that the monitors are marked as failing and the pool member is marked down, and the client sees a RST for "no pool member available".