Forum Discussion
Naresh_N
Nimbostratus
NimbostratusSSL handshake errors
Hi there,
Recently put TMOS version 12 into production and see following SSL handshake errors, none of which existed in version 10.2.3:
Nov 12 03:15:36 dc1lbc2p info tmm[11446]: 01260013:6: SSL Handshake failed for TCP 72.238.29.206:60819 -> x.x.x.x:443 Nov 12 03:15:55 dc1lbc2p info tmm[11446]: 01260013:6: SSL Handshake failed for TCP 96.241.137.52:50815 -> x.x.x.x:443 Nov 12 03:16:12 dc1lbc2p info tmm[11446]: 01260013:6: SSL Handshake failed for TCP 166.172.187.30:38119 -> x.x.x.x:443 Nov 12 03:16:32 dc1lbc2p warning tmm[11446]: 01260009:4: Connection error: hud_ssl_handler:1135: codec alert (20) Nov 12 03:16:32 dc1lbc2p info tmm[11446]: 01260013:6: SSL Handshake failed for TCP y.y.y.y:63127 -> z.z.z.z:443 Nov 12 03:18:53 dc1lbc2p warning tmm[11446]: 01260009:4: Connection error: ssl_hs_rxhello:7103: unsupported version (40)
Did ssldump and ssl debugs but can't figure it out. There are no low encryption ciphers being presented by clients. In fact I don't see any handshake errors in the packet captures. Its pretty baffling. Would be great if someone can throw some light. Techs at F5 haven't been able to figure it out either.
Thanks Naresh
Naresh_NKevin,
I reproduced this error on renegotiation. Could this be the cause for all those handshake errors:
openssl s_client -connect x.x.x.x:443 CONNECTED(00000003) depth=2 C = US, O = "thawte, Inc.", OU = Certification Services Division, OU = "(c) 2008 thawte, Inc. - For authorized use only", CN = thawte Primary Root CA - G3 verify return:1 depth=1 C = US, O = "thawte, Inc.", CN = thawte SHA256 SSL CA verify return:1 Removed bunch of stuff here SSL handshake has read 2983 bytes and written 442 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 82F80492D73C79776AC0FCEE3C8520FF237E79DB979BCBFFB62633FD0E15700F Session-ID-ctx: Master-Key: D36B850714E6F95E22DA3653F9410D7F10F8DE29A1BD889602DA27D28006D9258895C0777B361BAF6AE506B983CCF9F9 Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None Start Time: 1448044924 Timeout : 300 (sec) Verify return code: 0 (ok) --- R RENEGOTIATING 2283136:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:s3_pkt.c:1472:SSL alert number 40 2283136:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:s3_pkt.c:656:What do you think?
- Naresh_N
Nevermind, LTM logs throw a different error here.
Nov 20 10:46:46 dc1lbc2s warning tmm[8569]: 01260009:4: Connection error: ssl_hs_rxhello:6822: renegotiation disallowed (40) - Naresh_N
Kevin,
What went wrong here -
363 1 1448048245.1572 (0.0026) C>SV3.2(163) Handshake ClientHello Version 3.2 random[32]= 56 4f 76 75 17 79 bb e4 bb 1d 18 6d 65 65 f7 14 60 a5 de 1b 2c dc 2a d7 3d ee 8c d0 0e a4 83 e1 cipher suites TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA TLS_ECDH_RSA_WITH_AES_256_CBC_SHA TLS_DHE_RSA_WITH_AES_256_CBC_SHA TLS_DHE_DSS_WITH_AES_256_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA TLS_ECDH_RSA_WITH_AES_128_CBC_SHA TLS_DHE_RSA_WITH_AES_128_CBC_SHA TLS_DHE_DSS_WITH_AES_128_CBC_SHA TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA TLS_ECDHE_ECDSA_WITH_RC4_128_SHA TLS_ECDHE_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_RC4_128_SHA TLS_ECDH_ECDSA_WITH_RC4_128_SHA TLS_ECDH_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_RC4_128_MD5 TLS_EMPTY_RENEGOTIATION_INFO_SCSV compression methods NULL 363 2 1448048245.1584 (0.0011) S>CV3.2(87) Handshake ServerHello Version 3.2 random[32]= 52 c7 9d f6 0f e8 9e c4 39 2d 8e 51 49 ef b4 12 8a 3a 68 15 fe 7b a9 5c a7 de f9 e0 46 27 b0 20 session_id[32]= 1a b0 5c 0b ee 75 50 6f 02 78 ec a2 57 bd f8 f7 e3 72 9d 63 8f 53 a3 07 57 8d 9b 75 26 4d 48 07 cipherSuite TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA compressionMethod NULL 363 3 1448048245.1584 (0.0000) S>CV3.2(2488) Handshake Certificate Subject C=US ST=California L=Mountain View O=abc, Inc. OU=Network Operations CN=*.abc.net Issuer C=US O=thawte, Inc. CN=thawte SHA256 SSL CA Serial 49 a1 db 0d 32 5e a5 16 dd 0b 5c 71 eb ec f3 6a Extensions Extension: X509v3 Subject Alternative Name Extension: X509v3 Basic Constraints Extension: X509v3 Certificate Policies Extension: X509v3 Key Usage Critical Extension: X509v3 Authority Key Identifier Extension: X509v3 CRL Distribution Points Extension: X509v3 Extended Key Usage Extension: Authority Information Access Subject C=US O=thawte, Inc. CN=thawte SHA256 SSL CA Issuer C=US O=thawte, Inc. OU=Certification Services Division OU=(c) 2008 thawte, Inc. - For authorized use only CN=thawte Primary Root CA - G3 Serial 36 34 9e 18 c9 9c 26 69 b6 56 2e 6c e5 ad 71 32 Extensions Extension: Authority Information Access Extension: X509v3 Basic Constraints Critical Extension: X509v3 Certificate Policies Extension: X509v3 CRL Distribution Points Extension: X509v3 Key Usage Critical Extension: X509v3 Subject Alternative Name Extension: X509v3 Subject Key Identifier Extension: X509v3 Authority Key Identifier 363 4 1448048245.1584 (0.0000) S>CV3.2(331) Handshake ServerKeyExchange 363 5 1448048245.1584 (0.0000) S>CV3.2(4) Handshake ServerHelloDone 363 6 1448048245.1624 (0.0039) C>SV3.2(70) Handshake ClientKeyExchange 363 7 1448048245.1641 (0.0017) C>SV3.2(1) ChangeCipherSpec 363 8 1448048245.1642 (0.0000) C>SV3.2(64) Handshake 363 9 1448048245.1644 (0.0001) S>CV3.2(2) Alert level fatal value bad_record_mac 363 1448048245.1644 (0.0000) S>C TCP FIN 363 1448048245.1651 (0.0007) C>S TCP FIN - Kevin_Stewart
Employee
The first encrypted message sent by either party is the Finished message, seen here as the "Handshake" message at 363 8. The alert essentially means that the server (F5) was unable to decrypt this message and/or verify its mac (ie. signature). Unfortunately there's a few different reasons for this event to occur, including a bad client or server crypto implementation, bad server public or private key, and potentially any network issues that could have corrupted some part of the previous handshake.
Have you been able to isolate and reproduce this error?
- Naresh_N
Kevin,
This is not coming from a browser, its coming from a Java client. Not sure how to reproduce this, happens at random every few minutes.
- Kevin_Stewart
Hard to troubleshoot that sort of thing. Does the Java client log anything extra? Does it store the server's public key anywhere? Is it that this same Java client randomly fails across multiple tests, or fails every time?
- Naresh_N
I'll check if java client logs anything and if servers public key is cached (even if its cached, it doesn't change, should that matter?). There are multiple java clients failing randomly against 2 VIPs that they connect for ldap auth.
- Naresh_N
This one I can understand why it failed.
488 1 1448055944.6885 (0.0545) C>SV3.0(53) Handshake ClientHello Version 3.0 random[32]= 56 4f 94 87 6d 31 8e 91 b2 11 64 76 c0 89 64 73 7b eb 38 af 52 e8 83 22 6b 9a 9e 00 63 19 fb db cipher suites SSL_RSA_WITH_3DES_EDE_CBC_SHA SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA SSL_RSA_WITH_RC4_128_SHA SSL_RSA_WITH_RC4_128_MD5 SSL_EMPTY_RENEGOTIATION_INFO_SCSV compression methods NULL 488 2 1448055944.6887 (0.0001) S>CV3.0(2) Alert level fatal value handshake_failure 488 1448055944.6887 (0.0000) S>C TCP FIN- Kevin_StewartThis is failing because the client is attempting to negotiate SSLv3.
- Naresh_N
Here is another commonly failing handshake:
Captured 3 different streams from 3 clients, fails right after ServerHelloDone. Client sends finish, not sure why.
154 1 1448055928.8001 (0.1000) C>SV3.1(175) Handshake 154 2 1448055928.8013 (0.0011) S>CV3.3(91) Handshake 154 3 1448055928.8013 (0.0000) S>CV3.3(2488) Handshake 154 4 1448055928.8013 (0.0000) S>CV3.3(365) Handshake 154 5 1448055928.8013 (0.0000) S>CV3.3(4) Handshake ServerHelloDone 154 1448055928.9049 (0.1035) C>S TCP FIN 154 1448055928.9049 (0.0000) S>C TCP FIN 406 1 1448055941.9187 (0.2517) C>SV3.1(114) Handshake 406 2 1448055941.9196 (0.0009) S>CV3.1(80) Handshake 406 3 1448055941.9196 (0.0000) S>CV3.1(2488) Handshake 406 4 1448055941.9196 (0.0000) S>CV3.1(525) Handshake 406 5 1448055941.9196 (0.0000) S>CV3.1(4) Handshake ServerHelloDone 406 1448055942.2304 (0.3107) C>S TCP FIN 406 1448055942.2306 (0.0001) S>C TCP FIN 604 1 1448055950.8339 (1.0157) C>SV3.1(233) Handshake 604 2 1448055950.8351 (0.0012) S>CV3.3(91) Handshake 604 3 1448055950.8351 (0.0000) S>CV3.3(2488) Handshake 604 4 1448055950.8351 (0.0000) S>CV3.3(333) Handshake 604 5 1448055950.8351 (0.0000) S>CV3.3(4) Handshake ServerHelloDone 604 1448055951.8045 (0.9693) C>S TCP FIN 604 1448055951.8046 (0.0001) S>C TCP FIN - Kevin_Stewart
The client sends a ClientKeyExchange message right after the server's ServerHelloDone, which implies that the client is failing in some way to validate the server's certificate.
