New SSL Certificate update breaks F5 On-Demand

Okay, let's start with a quick rundown of the SSL handshake process (I'm simplifying it of course):

1. ClientHello - client initiates the SSL handshake and sends a list of capabilities (supported protocols, ciphers, extensions)

2. ServerHello - server responds to ClientHello with a chosen protocol and cipher, and reciprocates any extensions as required

3. Certificate - server immediately sends its certificate to the client, which the client must be able to the validate. For browsers, it usually means a pop-up nag screen that you can click past, but for some other user agents it may just fail right there if the trust can't be established. The trust is established by virtue of the client's ability to build an unbroken "chain" from the server's cert, up through its issuing authority, and up to the self-signed root CA.

4. Certificate Request - if mutual authentication is required the server sends this message and an optional list of "root hints". These root hints are the named and identifiers of the issuing CAs that the server will accepts client certificates from. In the BIG-IP client SSL profile this is handled by the Advertised Certificate Authorities selection. In the browser's context, if the root hints don't include an issuing CA that issued (any of) the client's certificate(s), then the browser either won't send a cert or won't prompt the user to select a cert if there's more than one.

5. ServerHelloDone - server tells client it's done

6. Certificate - this is the client sending the server its certificate, for which the server must be able to validate. In the BIG-IP client SSL profile this is handled by the Trusted Certificate Authorities selection and is a single CA or bundle of CAs that are needed to explicitly build an unbroken chain from the client's cert, up through its issuing authority, and up to the self-signed root CA. Unlike a browser, there's no nag screen. If the BIG-IP cannot build this trust chain, the certificate validation will fail.

7. ...the rest of the handshake... - not important to discuss here since it doesn't seem the problem you're seeing is getting past the above messages.

So the “Server rejected the supplied client certificate or one was not sent” message would seem to imply that the client either isn't sending a certificate (because of the advertised certificate authorities list), or the BIG-IP can't validate the client's certificate (because of the trusted certificate authorities list). The best way to validate when this is happening is with an SSLDUMP capture:

ssldump -AdNn -i 0.0 port 443 and host [IP of VIP] [and other optional filters]

In this capture you'll essentially see the above messages in roughly the same order. If it fails after the client's Certificate message, then you have reason to suspect a server side trust chain issue. If the client never sends a Certificate message, then you have reason to suspect a root hints issue. And if you do this capture in WireShark, or tcpdump and import into WireShark:

tcpdump -lnni 0.0 -Xs0 -w capture_file.pcap port 443 and host [IP of VIP]

you'll be able to see exactly what root hints the server (BIG-IP) is sending, and exactly what certificate the client is sending.

Give that a go and tell us what you find.