Persistence using source IP and source port

Source ports are not suitable for connection stickyness because clients will create many connections, each with a different port. SSL session IDs, same thing, the client will create many.

Source address should be used. You will see some imbalance, especially with a small number of connections but overall the problem should be negligible. The actual trouble may be that the downstram load balancer is confused by a source NAT. try disabling source NAT on both devices.