One Virtual Server, multiple ports and multiple Pools

You can achieve that by adding all server+port combinations to single pool and set a proper health monitor to it. The monitor will automatically disable/enable those pool members that stop/start to respond to specific service/port. Another approach is to create multiple pools (each for different set of ports) and put an irule to a VS, that will direct the traffic to respective pool based on the destination VIP/VS port.