have ssl and non ssl pools (or pool members) for a VS

You can use LTM policy or iRule to disable server SSL profile based on matching specific condition e.g. hostname. I would recommend you to use LTM policy for achieving this. While creating the LTM policy, you can match condition for the incoming request like hostname/URL/URI for which you want to disable server SSL and set disable server ssl action for it. After applying this LTM policy on the desired vServer, your requirement should be fulfilled.

Hope it helps!