SMTP ip address for VIP and relay

I'm told forward and reverse DNS needs to match to pass mail filters

This may be a policy set by your organization, but is not required by the technology alone. Most mail validation is handled by SPF records (a subset of the DNS type TXT record). It is common to have one or multiple IP addresses and DNS entries for inbound mail, but to use multiple separate IP addresses and DNS entries for outbound mail. With a properly configured SPF record, this can be accomplished.

That being said, the F5 builds a new connection using many different variables. We are specifically interested in the IP and TCP information. Inbound mail will initiate from an external address bound for our VIP using a TCP SYN packet first. When the mail server sends outbound mail, it will first establish a new connection on the server, sending a new TCP SYN packet to the load balancer. The F5 interprets this as a new connection, even though it appears similar to the already existing connection. Reference the TCP 3-way handshake (link on Wikipedia) for more information on how this works.

Is this a valid setup?

I mostly avoided this question because I'm not sure about the SNAT pool with a member as a VIP, suffice to say I would test it to see what happens. I would trust the F5 to keep the inbound and outbound connections separate.