F5 not routiing traffic through floating IP

Hard to say, depending what is your target solution.

If you would like to have client ACK and server SYN, ACK to be passed via F5 then you need to modify configuration of your FW.

As far as I understand it's FW that is sending client ACK directly to server. Instead of this it should send it to external Self IP of your F5 (one via which F5 can reach FW - if your VS is ForwardingIP type) or to VS IP (if your VS is PerformanceL4 type - in this case you will have to change Destination IP of VS to be in the same subnet as FW). Then all traffic between client and server will be processed by F5.

So flow should be like that:

ACK: client -> FW -> F5 192.168.x.x Self IP -> server

SYN, ACK: server -> F5 self IP in subnet used by server -> FW -> client - this one is already working

Of course for that to work your VS has to be configured to be enabled on F5 VLAN via which F5 is accessing FW.

For such configuration you don't need Loose Initiation/Close and you can as well use Standard type VS with Pool containing your SMTP server.

Or you can keep using current fastL4 VS (don't know if it's PerformanceL4 or Forwarding type).

Piotr