can nsx load balancer work with f5 for vm and physical server load balance

You wouldn't use both, pick one or the other. The NSX (Edge) load balancing functions are limited compared to what the F5 has to offer. You can of course integrate F5 into an NSX environment simply using VXLAN and VE.

You only need BIG-IQ if you want to make use of NSX's integrated workflow and service insertion etc.