F5 BigIP VIP listening on port 443 load balancing to IPs on different ports

On the F5, we have a VIP setup listening on port 443 . it's setup to load balance on IPs listening on two different ports. However when we call the VIP, we get a connection refused. We ran some traces and we see that instead of making calls on each of the ports, its calling on 443. We think we are missing port translation from 443 to the other ports.

Here is a scenario we are testing VIP foo.foo.com on port 443 is load balancing on boo1.abcd.com port 2121 and boo1.abcd.com port 2323 So when we call https://foo.foo.com/service instead of calling boo1.abcd.com on port 2121 or 2323, it calls boo1.abcd.com on port 443 instead

Can someone help figure what we may be missing in the F5 VIP setup to get this to work?