Mutual Auth Failing On Client Certificate Verification

The Trusted Certificate Authorities option is either a single certificate or text file bundle of certificates that represent the certifying authorities that the system uses to validate the client's certificate. This should be the complete chain of trust from the issuer of the client's cert, to that cert's issuer, and beyond until terminating at the self-signed root. Example:

client cert -> subordinate CA -> subordinate CA -> root CA    

It might have worked before with the self-signed local certs because you were providing a "closed loop" of trust. The Advertised Certificate Authorities option is a single certificate or text bundle of certificates that the system will use to "hint" to a certifying path. For example, let's say Verisign's root signs/issues two subordinate CAs which in turn sign/issue various client and server certs. The client may have certificates installed from both subordinates. The path hint tells the browser which path the server is expecting. To filter to only certs issued by the first subordinate, the Advertised Certificate Authorities option would ONLY contain this subordinate CA cert.

With respect to filtering to a subset of certificates within the same issuance, you would need to allow all for validation, and then post-filter with an iRule.