How to configure a virtual server with serverssl profiles to talk to both HTTP and HTTPS backends?

You can use an irule with LB_SELECTED event. Obtain information about which pool was selected via the LB::server command. Then all you need to do is Disable server-SSL, or enable it depending on your logic.

I would suggest to attach a Oneconnect profile to the VIP for this kind of use case, so that load balancing checks are performed on every request (instead of once at the start of connection) due to the influence that http keepalive can have.

You can use an irule with LB_SELECTED event. Obtain information about which pool was selected via the LB::server command. Then all you need to do is Disable server-SSL, or enable it depending on your logic.

I would suggest to attach a Oneconnect profile to the VIP for this kind of use case, so that load balancing checks are performed on every request (instead of once at the start of connection) due to the influence that http keepalive can have.

If I may expand, I'd make a few additional points.

1. Do you need SNI information applied to the server SSL profile(s)? Do you even really need multiple server SSL profiles? Most SSL sessions are client-initiated, so on the server side, the server SSL profile is responsible for initiating the SSL handshake to the server. In other words, it speaks first. The Server Name (SNI) field in the server SSL profile is used to inject a Server Name Indication extension in the F5's ClientHello message to the server. You'd only need this if the server actually required it, which they usually don't.

2. Switching and/or enabling/disabling server side attributes needs to happen before the server side context (SERVER_CONNECTED), so LB_SELECTED would work, but for what you're doing HTTP_REQUEST would probably be more appropriate. SO basically, assign a server SSL profile to the VIP, and selectively disable it based on the client side HTTP request. Something like this:

   when CLIENT_ACCEPTED {
       SSL::disable serverside
   }
   when HTTP_REQUEST {
       switch [string to lower [HTTP::host]] {
           "foo.company.com" { SSL::enable serverside }
       }
   }
   

3. If you're switching between different pool members during a single client connection, then you absolutely should enable OneConnect. But if the client is only going to a single pool member, then you don't it.

We assumed (maybe incorrectly but this does work) that the SNI name in the serverssl profile would be used to match up to what the backend "served" up and so validate the backend certificate.

It does not. The SNI is only used by the server, if at all needed.

2, so we are doing this for a cluster where we programmatically add pools/serverssl profiles/certs etc and so we could need to keep a track of all hosts we added serverssl profiles for. So one suggestion we had was to create a datagroup that contains that set of host names and key enabling ssl serverside off of that

The data group option is probably the best option. You simply need to know, based on the Host header, whether or not to use server side SSL, correct? You could also include the pool in that data group if you're also switching between pools.