Forum Discussion
Naresh_N
Nimbostratus
NimbostratusSSL handshake errors
Hi there,
Recently put TMOS version 12 into production and see following SSL handshake errors, none of which existed in version 10.2.3:
Nov 12 03:15:36 dc1lbc2p info tmm[11446]: 01260013:6: SSL Handshake failed for TCP 72.238.29.206:60819 -> x.x.x.x:443 Nov 12 03:15:55 dc1lbc2p info tmm[11446]: 01260013:6: SSL Handshake failed for TCP 96.241.137.52:50815 -> x.x.x.x:443 Nov 12 03:16:12 dc1lbc2p info tmm[11446]: 01260013:6: SSL Handshake failed for TCP 166.172.187.30:38119 -> x.x.x.x:443 Nov 12 03:16:32 dc1lbc2p warning tmm[11446]: 01260009:4: Connection error: hud_ssl_handler:1135: codec alert (20) Nov 12 03:16:32 dc1lbc2p info tmm[11446]: 01260013:6: SSL Handshake failed for TCP y.y.y.y:63127 -> z.z.z.z:443 Nov 12 03:18:53 dc1lbc2p warning tmm[11446]: 01260009:4: Connection error: ssl_hs_rxhello:7103: unsupported version (40)
Did ssldump and ssl debugs but can't figure it out. There are no low encryption ciphers being presented by clients. In fact I don't see any handshake errors in the packet captures. Its pretty baffling. Would be great if someone can throw some light. Techs at F5 haven't been able to figure it out either.
Thanks Naresh
- Kevin_Stewart
Employee
Could it be because I am using SHA-256 certificate chain?
Maybe, but if that was it I would think it would fail every time. And not all of the errors are the same, and I assume not from the same clients.
Naresh_NCould it be because I am using SHA-256 certificate chain?
- Kevin_Stewart
The client sends a ClientKeyExchange message right after the server's ServerHelloDone, which implies that the client is failing in some way to validate the server's certificate.
- Naresh_N
Here is another commonly failing handshake:
Captured 3 different streams from 3 clients, fails right after ServerHelloDone. Client sends finish, not sure why.
154 1 1448055928.8001 (0.1000) C>SV3.1(175) Handshake 154 2 1448055928.8013 (0.0011) S>CV3.3(91) Handshake 154 3 1448055928.8013 (0.0000) S>CV3.3(2488) Handshake 154 4 1448055928.8013 (0.0000) S>CV3.3(365) Handshake 154 5 1448055928.8013 (0.0000) S>CV3.3(4) Handshake ServerHelloDone 154 1448055928.9049 (0.1035) C>S TCP FIN 154 1448055928.9049 (0.0000) S>C TCP FIN 406 1 1448055941.9187 (0.2517) C>SV3.1(114) Handshake 406 2 1448055941.9196 (0.0009) S>CV3.1(80) Handshake 406 3 1448055941.9196 (0.0000) S>CV3.1(2488) Handshake 406 4 1448055941.9196 (0.0000) S>CV3.1(525) Handshake 406 5 1448055941.9196 (0.0000) S>CV3.1(4) Handshake ServerHelloDone 406 1448055942.2304 (0.3107) C>S TCP FIN 406 1448055942.2306 (0.0001) S>C TCP FIN 604 1 1448055950.8339 (1.0157) C>SV3.1(233) Handshake 604 2 1448055950.8351 (0.0012) S>CV3.3(91) Handshake 604 3 1448055950.8351 (0.0000) S>CV3.3(2488) Handshake 604 4 1448055950.8351 (0.0000) S>CV3.3(333) Handshake 604 5 1448055950.8351 (0.0000) S>CV3.3(4) Handshake ServerHelloDone 604 1448055951.8045 (0.9693) C>S TCP FIN 604 1448055951.8046 (0.0001) S>C TCP FIN - Naresh_N
This one I can understand why it failed.
488 1 1448055944.6885 (0.0545) C>SV3.0(53) Handshake ClientHello Version 3.0 random[32]= 56 4f 94 87 6d 31 8e 91 b2 11 64 76 c0 89 64 73 7b eb 38 af 52 e8 83 22 6b 9a 9e 00 63 19 fb db cipher suites SSL_RSA_WITH_3DES_EDE_CBC_SHA SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA SSL_RSA_WITH_RC4_128_SHA SSL_RSA_WITH_RC4_128_MD5 SSL_EMPTY_RENEGOTIATION_INFO_SCSV compression methods NULL 488 2 1448055944.6887 (0.0001) S>CV3.0(2) Alert level fatal value handshake_failure 488 1448055944.6887 (0.0000) S>C TCP FIN- Kevin_StewartThis is failing because the client is attempting to negotiate SSLv3.
- Naresh_N
I'll check if java client logs anything and if servers public key is cached (even if its cached, it doesn't change, should that matter?). There are multiple java clients failing randomly against 2 VIPs that they connect for ldap auth.
- Kevin_Stewart
Hard to troubleshoot that sort of thing. Does the Java client log anything extra? Does it store the server's public key anywhere? Is it that this same Java client randomly fails across multiple tests, or fails every time?
- Naresh_N
Kevin,
This is not coming from a browser, its coming from a Java client. Not sure how to reproduce this, happens at random every few minutes.
- Kevin_Stewart
The first encrypted message sent by either party is the Finished message, seen here as the "Handshake" message at 363 8. The alert essentially means that the server (F5) was unable to decrypt this message and/or verify its mac (ie. signature). Unfortunately there's a few different reasons for this event to occur, including a bad client or server crypto implementation, bad server public or private key, and potentially any network issues that could have corrupted some part of the previous handshake.
Have you been able to isolate and reproduce this error?
- Naresh_N
Kevin,
What went wrong here -
363 1 1448048245.1572 (0.0026) C>SV3.2(163) Handshake ClientHello Version 3.2 random[32]= 56 4f 76 75 17 79 bb e4 bb 1d 18 6d 65 65 f7 14 60 a5 de 1b 2c dc 2a d7 3d ee 8c d0 0e a4 83 e1 cipher suites TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA TLS_ECDH_RSA_WITH_AES_256_CBC_SHA TLS_DHE_RSA_WITH_AES_256_CBC_SHA TLS_DHE_DSS_WITH_AES_256_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA TLS_ECDH_RSA_WITH_AES_128_CBC_SHA TLS_DHE_RSA_WITH_AES_128_CBC_SHA TLS_DHE_DSS_WITH_AES_128_CBC_SHA TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA TLS_ECDHE_ECDSA_WITH_RC4_128_SHA TLS_ECDHE_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_RC4_128_SHA TLS_ECDH_ECDSA_WITH_RC4_128_SHA TLS_ECDH_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_RC4_128_MD5 TLS_EMPTY_RENEGOTIATION_INFO_SCSV compression methods NULL 363 2 1448048245.1584 (0.0011) S>CV3.2(87) Handshake ServerHello Version 3.2 random[32]= 52 c7 9d f6 0f e8 9e c4 39 2d 8e 51 49 ef b4 12 8a 3a 68 15 fe 7b a9 5c a7 de f9 e0 46 27 b0 20 session_id[32]= 1a b0 5c 0b ee 75 50 6f 02 78 ec a2 57 bd f8 f7 e3 72 9d 63 8f 53 a3 07 57 8d 9b 75 26 4d 48 07 cipherSuite TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA compressionMethod NULL 363 3 1448048245.1584 (0.0000) S>CV3.2(2488) Handshake Certificate Subject C=US ST=California L=Mountain View O=abc, Inc. OU=Network Operations CN=*.abc.net Issuer C=US O=thawte, Inc. CN=thawte SHA256 SSL CA Serial 49 a1 db 0d 32 5e a5 16 dd 0b 5c 71 eb ec f3 6a Extensions Extension: X509v3 Subject Alternative Name Extension: X509v3 Basic Constraints Extension: X509v3 Certificate Policies Extension: X509v3 Key Usage Critical Extension: X509v3 Authority Key Identifier Extension: X509v3 CRL Distribution Points Extension: X509v3 Extended Key Usage Extension: Authority Information Access Subject C=US O=thawte, Inc. CN=thawte SHA256 SSL CA Issuer C=US O=thawte, Inc. OU=Certification Services Division OU=(c) 2008 thawte, Inc. - For authorized use only CN=thawte Primary Root CA - G3 Serial 36 34 9e 18 c9 9c 26 69 b6 56 2e 6c e5 ad 71 32 Extensions Extension: Authority Information Access Extension: X509v3 Basic Constraints Critical Extension: X509v3 Certificate Policies Extension: X509v3 CRL Distribution Points Extension: X509v3 Key Usage Critical Extension: X509v3 Subject Alternative Name Extension: X509v3 Subject Key Identifier Extension: X509v3 Authority Key Identifier 363 4 1448048245.1584 (0.0000) S>CV3.2(331) Handshake ServerKeyExchange 363 5 1448048245.1584 (0.0000) S>CV3.2(4) Handshake ServerHelloDone 363 6 1448048245.1624 (0.0039) C>SV3.2(70) Handshake ClientKeyExchange 363 7 1448048245.1641 (0.0017) C>SV3.2(1) ChangeCipherSpec 363 8 1448048245.1642 (0.0000) C>SV3.2(64) Handshake 363 9 1448048245.1644 (0.0001) S>CV3.2(2) Alert level fatal value bad_record_mac 363 1448048245.1644 (0.0000) S>C TCP FIN 363 1448048245.1651 (0.0007) C>S TCP FIN
