Forum Discussion
Naresh_N
Nimbostratus
NimbostratusSSL handshake errors
Hi there,
Recently put TMOS version 12 into production and see following SSL handshake errors, none of which existed in version 10.2.3:
Nov 12 03:15:36 dc1lbc2p info tmm[11446]: 01260013:6: SSL Handshake failed for TCP 72.238.29.206:60819 -> x.x.x.x:443 Nov 12 03:15:55 dc1lbc2p info tmm[11446]: 01260013:6: SSL Handshake failed for TCP 96.241.137.52:50815 -> x.x.x.x:443 Nov 12 03:16:12 dc1lbc2p info tmm[11446]: 01260013:6: SSL Handshake failed for TCP 166.172.187.30:38119 -> x.x.x.x:443 Nov 12 03:16:32 dc1lbc2p warning tmm[11446]: 01260009:4: Connection error: hud_ssl_handler:1135: codec alert (20) Nov 12 03:16:32 dc1lbc2p info tmm[11446]: 01260013:6: SSL Handshake failed for TCP y.y.y.y:63127 -> z.z.z.z:443 Nov 12 03:18:53 dc1lbc2p warning tmm[11446]: 01260009:4: Connection error: ssl_hs_rxhello:7103: unsupported version (40)
Did ssldump and ssl debugs but can't figure it out. There are no low encryption ciphers being presented by clients. In fact I don't see any handshake errors in the packet captures. Its pretty baffling. Would be great if someone can throw some light. Techs at F5 haven't been able to figure it out either.
Thanks Naresh
Naresh_NNevermind, LTM logs throw a different error here.
Nov 20 10:46:46 dc1lbc2s warning tmm[8569]: 01260009:4: Connection error: ssl_hs_rxhello:6822: renegotiation disallowed (40)- Naresh_N
Kevin,
I reproduced this error on renegotiation. Could this be the cause for all those handshake errors:
openssl s_client -connect x.x.x.x:443 CONNECTED(00000003) depth=2 C = US, O = "thawte, Inc.", OU = Certification Services Division, OU = "(c) 2008 thawte, Inc. - For authorized use only", CN = thawte Primary Root CA - G3 verify return:1 depth=1 C = US, O = "thawte, Inc.", CN = thawte SHA256 SSL CA verify return:1 Removed bunch of stuff here SSL handshake has read 2983 bytes and written 442 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 82F80492D73C79776AC0FCEE3C8520FF237E79DB979BCBFFB62633FD0E15700F Session-ID-ctx: Master-Key: D36B850714E6F95E22DA3653F9410D7F10F8DE29A1BD889602DA27D28006D9258895C0777B361BAF6AE506B983CCF9F9 Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None Start Time: 1448044924 Timeout : 300 (sec) Verify return code: 0 (ok) --- R RENEGOTIATING 2283136:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:s3_pkt.c:1472:SSL alert number 40 2283136:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:s3_pkt.c:656:What do you think?
- Naresh_N
Kevin,
Failures are coming from random clients over the Internet. I haven't been able to reproduce and so my captures won't show the error. I'm not sure if there is a way to create it.
Naresh
- Kevin_Stewart
Employee
Okay, so if you're not having any issues getting to the application via browsers, I'd simply run a client side capture to see if any elements of that application are failing. It doesn't really make any sense that the SSL handshakes you're looking at are failing where they are, but that you're still able to get to the application. Something's missing in this equation.
- Naresh_N
Not found literally with "chrome://net-internals/hsts". Not a 404 error. I wasn't able to reproduce a connection failure with any of the browsers while going to the web site. I just see handshake errors.
- Kevin_Stewart
I get "Not found" in chrome. I was never able to reproduce a not available site in any of the browsers. I have only seen SSL handshake errors in logs.
A "Not Found" as in a 404 response? A 404 is an application layer error, which implies a successful SSL handshake. Can you do a client side capture (HTTPWatch, Fiddler, etc.) and look for exactly what the client is asking for and if any of those requests are getting answered. It may be that some of the resource requests are failing, but not all.
Do you get the same error in all browsers?
- Naresh_N
I get "Not found" in chrome. I was never able to reproduce a not available site in any of the browsers. I have only seen SSL handshake errors in logs.
- Kevin_Stewart
A couple of things to note:
-
You still have a few different client connections going on here. You can tell by the first number on each line (ex. 30, 31, 13, 254). I'll focus on 31 and 254 as they're the most complete captures.
-
The "Critical" you're seeing is an X.509 extension property. In most cases a client or server in receipt of a peer's certificate can ignore X.509 extensions that it doesn't understand, except in the case of critical extensions. It's quite common for the Key Usage and Basic Constraints extensions to be marked Critical.
-
Tcpdump and ssldump can sometimes display packets out of order. In this case though, the very next message should be a ClientKeyExchange from the client, so we can be pretty sure here that the client is the one with the issue.
-
I'm not definitively saying that HSTS is the issue here, but in a real-world scenario HSTS will prevent access if the browser can't trust the presented certificate. Because we know the HSTS header is/was present, it's a "low hanging fruit" so to speak. Have you tried this with a few different browsers (ie. Internet Explorer, Chrome, FireFox, etc.)? If it doesn't work in Chrome or Opera, go to chrome://net-internals/hsts and query for the domain there to see if it's in the list. If it isn't, then HSTS isn't the problem.
-
In any case, it appears from everything I've observed so far that the client is indeed the problem. If a cURL client (ignoring certificate trust) can access the site, but a browser can't, it almost certainly has something to do with certificate trust. Can you test with a new client SSL profile? Or better, the default clientssl profile (assuming HSTS isn't involved).
-
- Naresh_N
Ok. I disabled hsts and waited for 500 seconds and more, still got SSL handshake errors. Following are 2 examples when hsts was removed. Notice the "critical" in the middle of handshake and abrupt FINs sent by client and server.
New TCP connection 31: 70.114.201.226(55098) <-> x.x.x.x(443) 30 6 0.2040 (0.1128) C>SV3.3(70) Handshake ClientKeyExchange 30 7 0.2040 (0.0000) C>SV3.3(1) ChangeCipherSpec 30 8 0.2040 (0.0000) C>SV3.3(64) Handshake 30 9 0.2053 (0.0013) S>CV3.3(1) ChangeCipherSpec 30 10 0.2053 (0.0000) S>CV3.3(64) Handshake 31 1 0.1392 (0.1392) C>SV3.3(184) Handshake ClientHello Version 3.3 random[32]= 56 4c 20 92 37 cd ef 90 aa 66 36 f3 aa 9f be 9e d2 96 ac d6 6f 3f ee 4e 39 a6 d2 75 a3 c3 f5 82 cipher suites TLS_EMPTY_RENEGOTIATION_INFO_SCSV TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA TLS_ECDHE_ECDSA_WITH_RC4_128_SHA TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA TLS_ECDHE_RSA_WITH_RC4_128_SHA TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA TLS_ECDH_ECDSA_WITH_RC4_128_SHA TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA TLS_ECDH_RSA_WITH_AES_128_CBC_SHA TLS_ECDH_RSA_WITH_AES_256_CBC_SHA TLS_ECDH_RSA_WITH_RC4_128_SHA TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA256 TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_RC4_128_MD5 TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 TLS_DHE_RSA_WITH_AES_128_CBC_SHA TLS_DHE_RSA_WITH_AES_256_CBC_SHA TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA compression methods NULL 31 2 0.1403 (0.0010) S>CV3.3(91) Handshake ServerHello Version 3.3 random[32]= 67 f0 29 68 55 5a a3 e9 5d 9d 2d 11 4b fe 44 7d ba a8 3c c9 23 48 60 d7 63 ad b6 61 1e 53 c3 c9 session_id[32]= c6 6c e9 26 5b f1 9d 4a 2f b5 ba 33 42 b9 c4 53 c4 0a 4f 40 29 f2 9b 2c 65 6b ac e9 12 49 14 63 cipherSuite TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 compressionMethod NULL 31 3 0.1403 (0.0000) S>CV3.3(2488) Handshake Certificate Subject C=US ST=California L=Mountain View O=abc, Inc. OU=Network Operations CN=*.abc.net Issuer C=US O=thawte, Inc. CN=thawte SHA256 SSL CA Serial 49 a1 db 0d 32 5e a5 16 dd 0b 5c 71 eb ec f3 6a Extensions Extension: X509v3 Subject Alternative Name Extension: X509v3 Basic Constraints Extension: X509v3 Certificate Policies Extension: X509v3 Key Usage Critical Extension: X509v3 Authority Key Identifier Extension: X509v3 CRL Distribution Points Extension: X509v3 Extended Key Usage Extension: Authority Information Access Subject C=US O=thawte, Inc. CN=thawte SHA256 SSL CA Issuer C=US O=thawte, Inc. OU=Certification Services Division OU=(c) 2008 thawte, Inc. - For authorized use only CN=thawte Primary Root CA - G3 Serial 36 34 9e 18 c9 9c 26 69 b6 56 2e 6c e5 ad 71 32 Extensions Extension: Authority Information Access Extension: X509v3 Basic Constraints Critical Extension: X509v3 Certificate Policies Extension: X509v3 CRL Distribution Points Extension: X509v3 Key Usage Critical Extension: X509v3 Subject Alternative Name Extension: X509v3 Subject Key Identifier Extension: X509v3 Authority Key Identifier 31 4 0.1404 (0.0000) S>CV3.3(333) Handshake ServerKeyExchange 31 5 0.1404 (0.0000) S>CV3.3(4) Handshake ServerHelloDone 30 11 0.3403 (0.1350) C>SV3.3(544) application_data 30 12 0.3436 (0.0032) S>CV3.3(1456) application_data 30 13 0.3436 (0.0000) S>CV3.3(1488) application_data 30 14 0.3436 (0.0000) S>CV3.3(1488) application_data 30 15 0.3436 (0.0000) S>CV3.3(144) application_data 30 16 0.3441 (0.0004) S>CV3.3(1488) application_data 30 17 0.4304 (0.0862) S>CV3.3(1488) application_data 30 18 0.4304 (0.0000) S>CV3.3(1488) application_data 30 19 0.4304 (0.0000) S>CV3.3(1488) application_data 30 20 0.4304 (0.0000) S>CV3.3(1488) application_data 30 21 0.4304 (0.0000) S>CV3.3(1488) application_data 30 22 0.4304 (0.0000) S>CV3.3(1040) application_data 31 0.7382 (0.5978) C>S TCP FIN 31 0.7383 (0.0000) S>C TCP FIN 13 11 9.1953 (9.1702) C>SV3.3(448) application_data 13 12 9.1978 (0.0024) S>CV3.3(1472) application_data 13 13 9.1978 (0.0000) S>CV3.3(1520) application_data 13 14 9.1978 (0.0000) S>CV3.3(1520) application_data 13 15 9.1978 (0.0000) S>CV3.3(176) application_data 13 16 9.1980 (0.0001) S>CV3.3(1520) application_data 13 17 9.1980 (0.0000) S>CV3.3(1520) application_data 13 18 9.1981 (0.0001) S>CV3.3(1520) application_data 13 19 9.1981 (0.0000) S>CV3.3(1520) application_data 13 20 9.1982 (0.0001) S>CV3.3(1520) application_data 13 21 9.2086 (0.0103) S>CV3.3(1520) application_data 13 22 9.2098 (0.0011) S>CV3.3(1328) application_data New TCP connection 254: 81.169.237.146(49648) <-> x.x.x.x(443) 254 1 0.1686 (0.1686) C>SV3.1(160) Handshake ClientHello Version 3.1 random[32]= 4f e0 34 c6 b9 14 78 a8 d4 5c c8 bd 85 9c 4a 75 03 aa b4 69 4d e4 8a 1f ff 64 0b 25 e9 07 33 dd cipher suites TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA TLS_DHE_RSA_WITH_AES_128_CBC_SHA TLS_DHE_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA compression methods NULL 254 2 0.1697 (0.0010) S>CV3.1(87) Handshake ServerHello Version 3.1 random[32]= 42 31 2f 60 10 8f 42 75 ab ed 8a d9 03 32 16 7e b5 6e 06 64 62 9c 1e ec d5 fd 40 58 49 73 18 eb session_id[32]= c3 46 ec 69 a6 db 70 1d cb ae a2 bc 2f c2 39 44 6c 11 05 cf b4 d9 76 1b 2a 50 89 36 3f 92 c9 94 cipherSuite TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA compressionMethod NULL 254 3 0.1697 (0.0000) S>CV3.1(2488) Handshake Certificate Subject C=US ST=California L=Mountain View O=abc, Inc. OU=Network Operations CN=*.abc.net Issuer C=US O=thawte, Inc. CN=thawte SHA256 SSL CA Serial 49 a1 db 0d 32 5e a5 16 dd 0b 5c 71 eb ec f3 6a Extensions Extension: X509v3 Subject Alternative Name Extension: X509v3 Basic Constraints Extension: X509v3 Certificate Policies Extension: X509v3 Key Usage Critical Extension: X509v3 Authority Key Identifier Extension: X509v3 CRL Distribution Points Extension: X509v3 Extended Key Usage Extension: Authority Information Access Subject C=US O=thawte, Inc. CN=thawte SHA256 SSL CA Issuer C=US O=thawte, Inc. OU=Certification Services Division OU=(c) 2008 thawte, Inc. - For authorized use only CN=thawte Primary Root CA - G3 Serial 36 34 9e 18 c9 9c 26 69 b6 56 2e 6c e5 ad 71 32 Extensions Extension: Authority Information Access Extension: X509v3 Basic Constraints Critical Extension: X509v3 Certificate Policies Extension: X509v3 CRL Distribution Points Extension: X509v3 Key Usage Critical Extension: X509v3 Subject Alternative Name Extension: X509v3 Subject Key Identifier Extension: X509v3 Authority Key Identifier 254 4 0.1697 (0.0000) S>CV3.1(331) Handshake ServerKeyExchange 254 5 0.1697 (0.0000) S>CV3.1(4) Handshake ServerHelloDone 69 64.4153 (59.9118) S>C TCP FIN 69 64.4152 C>S TCP FIN 225 11 10.0037 (9.8402) C>SV3.1(752) application_data 225 12 10.0063 (0.0025) S>CV3.1(480) application_data 254 0.3390 (0.1692) C>S TCP FIN 254 0.3391 (0.0000) S>C TCP FIN - Kevin_Stewart
I should mention now, for the sake of completeness, that it's a security best practice to leave that header enabled. Once you've solved the trust issues without the header, you'll want to turn it back on.
