ssl_shim_vfycerterr:4539: application verification failure

am_gli_287451 · ‎07-Dec-2017

Hi,

after the Announcement regarding RSA vulnerability two weeks ago, we updated one of our BigIP from 11.5.3 to 12.1.2 HF2. We still have another F5 with 11.5.3 and the same application in place.

Important aspects:

HTTPS-VS, simple clientssl and an irule
clientssl profile -> Client Authentication: set to ignore, but the Trusted CA/Advertised CA are set to a bundle (XYZ)

there's an irule that checks for the URI, if it is /admin, following is done:

SSL::session invalidate
SSL::authenticate always
SSL::authenticate depth 9
SSL::cert mode require
SSL::renegotiate enable
SSL::renegotiate

there are two different types of client certificates: with SHA1 and with SHA256.
cipher string on both F5s is: DEFAULT:-TLSv1_1:-TLSv1:-SSLv3:-SSLv2:-RC4:-DES:-3DES

So the VS doesn't require a client-certificate by default, except for /admin. If /admin is requested, a SSL-renegotiation takes place. The client certificate is now "required", checked against XYZ bundle, and additionally checked against a data group of serial numbers.

Now the issue:

with v11.5.3 it worked properly for all certificates/browsers, and still works on the test-F5
since the update to v12.1.2 HF2, the SHA-1 certificates don't work any more, but SHA-256 do
with IE11 it works when disabling sslv2/v3 in the options
with IE11 it doesn't work with SHA1 when enabling sslv2/v3 in the options
in IE11 in both cases the same Cipher Suite is used
in any of the cases the client is asked to choose the certificate and enter the PIN (so the renegotiate part works)
Following error occurs in LTM log: Connection error: ssl_shim_vfycerterr:4539: application verification failure (46)

Does anyone have an idea where or what to check? At first I assumed different Cipher settings that have effect, because the default ciphers are different between v11/v12. But when I adapted the string to match exactly, it still didn't work. Now I assume that some other default settings in the SSL-profile or the handling of the SHA-1 certificates may have changed in some way.

But the strange thing is this behavior of IE11 with sslv2/sslv3 - normally it should only change the supported ciphers of the client/browser. But in both cases the same one is used and still works.

I also thought, maybe it is a browser issue (Chrome / SHA-1 support) - but since it works fine when connecting to an v11 F5, that shouldn't be the case.

Thanks in advance!

Andras_Kis-Szab · ‎13-Aug-2018

I assume that your client cert is not suitable for client authentication, that cert usage is missing from the cert. You might have a non-reputation only client cert.

am_gli · ‎13-Aug-2018

Thanks for the reply, the issue was resolved a few months ago but I forgot to update it here.

After some in-depth troubleshooting together with the support, we figured out that the issue was the "Signature Hash Algorithm".

In 11.5.3 the proposed algorithms were SHA1 (0x201-0x203) SHA2-256 (0x401-0x403) and SHA2-384 (0x501-503) - in that order. Browser accepted the first proposed one (SHA1) and proceeded properly to talk to the middleware and presented the client certificate.

With 12.1.2, the default list of the Algorithms changed - now SHA256/384/512 were all placed first and SHA1 came last. The browser negotiated SHA256 as the algorithm to use, talked to the middleware, but middleware said that this is not supported with the SHA-1 cards and didn't provide the certificate for the authentication.

Unfortunately neither the browser, nor the middleware came up with a proper error message...

After figuring that out, we manually forced SHA1 as the only algorithm to use, now it works with both smartcard types.

Andras_Kis-Szab · ‎13-Aug-2018

Oh, this issue has a kb article since that. I'll try to downgrade as well. The default is still any at 13.1.

Andras_Kis-Sza1 · ‎13-Aug-2018