Mutual Auth Failing On Client Certificate Verification

There is only one cert with chain, not 3 certificates one for each client cert -> subordinate CA -> root CA. Is this what you are suggesting that I do?

The F5 just needs to be able to validate the client's cert based on an explicitly defined trust list. If the client cert is signed by a subordinate CA, then that CA cert and its issuer, up to the root, should be in the Trusted Authorities bundle. If the client cert is signed by a root CA, then only that CA cert need be in the bundle.

Letting clients through by establishing an SSL connection to be validated later by another rule seems sub-optimal. At the end of the day, it seems like signing client certs with a non-public authority is our best bet. That way you can be sure that only the certs you sign have access to the service. Thoughts?

It really depends on who you are, what you're providing, and who it's being provided to. If you're a commercial or government entity trying to secure access to an application with client certificates, it may not make sense to also require a non-public authority and the overhead of its maintenance and distribution. For others who want more control, it might absolutely make sense. If you're getting your certs from a public CA, there's not much you can do other than filter after validation. This isn't an F5 limitation, but rather an inherent quality of the technology (PKI).

I'd also probably argue that access to a service shouldn't be granted based solely on the existence and validity of a client certificate. If you're talking about smart cards or other hardware tokens that provides at least two factors of authentication, then you might be able to get away with it, but generally speaking a software-based certificate provides an identity assertion and non-repudiation. You should still make sure the identity asserted by the cert is a valid user in a system and has access to use the service - another filtering function that happens post certificate validation.