LTM networking issue:

Do you have Loose Initiation enabled on the forwarding VS? If so (or maybe even if not) you could look into the VS attribute "Source Port: Preserve Strict" and/or the global option "VLAN-keyed connections".

I can't possibly tell if this is the reason for your problems. But your description sounds vaguely familiar to an issue I had once, where SYNACK Responses (received on a different VLAN than the original SYN, with VLAN-keyed Connections disabled at the time) were treated as new connections, their source port (which was actually the destination port of the SYN) was re-mapped and therefore the TCP handshake failed.