Forum Discussion

Altostratus

Jul 05, 2016

Solved

BIG-IP Proxy SSL 12.1 Handshake Failure

I set up SSL Proxy in order to do client certificate authentication on my IIS web server on LTM 12.1 firmware. The setup is working fine on Firefox version 43, IE 10 and OpenSSL but it fails on Chrome 51, Firefox 47 and IE 11.

I've captured the packets. Clients use TLS1 or TLS1.2 using the same ciphersuite of TLS_RSA_WITH_AES_256_CBC_SHA (0x0035), the same process takes place for the passing and failing cases.

Client Hello
Server Hello, Certificate, Server Hello Done
Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message 4. 4.1 Either Server sends Change Cipher Spec and then Application Data gets transfered Or 4.2 The server sends Alert level: Fatal, Descrition: Handshake Failure

So I suspect the BIG-IP fails to decrypt the handshake sent by the client in some cases but I can't figure out why because there's nothing different between failing and passing tests.

ssldump using Firefox 47 (Fails):

    New TCP connection 1: 192.168.100.125(55041) <-> 192.168.100.231(443)

1 1  0.0027 (0.0027)  C>S  Handshake

      ClientHello

        Version 3.3 

        cipher suites

        TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

        Unknown value 0xcca9

        Unknown value 0xcca8

        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA

        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

        TLS_DHE_RSA_WITH_AES_128_CBC_SHA

        TLS_DHE_RSA_WITH_AES_256_CBC_SHA

        TLS_RSA_WITH_AES_128_CBC_SHA

        TLS_RSA_WITH_AES_256_CBC_SHA

        TLS_RSA_WITH_3DES_EDE_CBC_SHA

        compression methods

                  NULL

1 2  0.0033 (0.0005)  S>C  Handshake

      ServerHello

        Version 3.3 

        session_id[32]=

          d9 0a 00 00 3e 11 22 ac e2 c2 00 f5 9a 41 35 53 

          43 6a 9e a5 e0 26 32 e4 f8 38 2e ca 72 3c fb 93 

        cipherSuite         TLS_RSA_WITH_AES_256_CBC_SHA

        compressionMethod                   NULL

      Certificate

      ServerHelloDone

1 3  0.0185 (0.0151)  C>S  Handshake

      ClientKeyExchange

1 4  0.0185 (0.0000)  C>S  ChangeCipherSpec

1 5  0.0185 (0.0000)  C>S  Handshake

1 6  0.0196 (0.0011)  S>C  Alert

    level           fatal

    value           handshake_failure

1    0.0197 (0.0000)  S>C  TCP FIN

1    0.0205 (0.0008)  C>S  TCP FIN

New TCP connection 2: 192.168.100.125(55042) <-> 192.168.100.231(443)

2 1  0.0005 (0.0005)  C>S  Handshake

      ClientHello

        Version 3.2 

        cipher suites

        Unknown value 0x5600

        Unknown value 0xcca9

        Unknown value 0xcca8

        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA

        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

        TLS_DHE_RSA_WITH_AES_128_CBC_SHA

        TLS_DHE_RSA_WITH_AES_256_CBC_SHA

        TLS_RSA_WITH_AES_128_CBC_SHA

        TLS_RSA_WITH_AES_256_CBC_SHA

        TLS_RSA_WITH_3DES_EDE_CBC_SHA

        compression methods

                  NULL

2 2  0.0010 (0.0005)  S>C  Handshake

      ServerHello

        Version 3.2 

        session_id[32]=

          85 48 00 00 8f 2a ae 80 b8 d7 e9 e2 47 c0 15 4e 

          e8 af 69 6f 2d b9 b8 d6 ed d5 29 3c a3 a3 44 b3 

        cipherSuite         TLS_RSA_WITH_AES_256_CBC_SHA

        compressionMethod                   NULL

      Certificate

      ServerHelloDone

2 3  0.0145 (0.0134)  C>S  Handshake

      ClientKeyExchange

2 4  0.0145 (0.0000)  C>S  ChangeCipherSpec

2 5  0.0145 (0.0000)  C>S  Handshake

2 6  0.0158 (0.0013)  S>C  Alert

    level           fatal

    value           handshake_failure

2    0.0158 (0.0000)  S>C  TCP FIN

2    0.0162 (0.0003)  C>S  TCP FIN

New TCP connection 3: 192.168.100.125(55043) <-> 192.168.100.231(443)

3 1  0.0005 (0.0005)  C>S  Handshake

      ClientHello

        Version 3.1 

        cipher suites

        Unknown value 0x5600

        Unknown value 0xcca9

        Unknown value 0xcca8

        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA

        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

        TLS_DHE_RSA_WITH_AES_128_CBC_SHA

        TLS_DHE_RSA_WITH_AES_256_CBC_SHA

        TLS_RSA_WITH_AES_128_CBC_SHA

        TLS_RSA_WITH_AES_256_CBC_SHA

        TLS_RSA_WITH_3DES_EDE_CBC_SHA

        compression methods

                  NULL

3 2  0.0010 (0.0004)  S>C  Handshake

      ServerHello

        Version 3.1 

        session_id[32]=

          aa 41 00 00 04 82 07 3f ed 35 96 49 e2 c5 ba 79 

          f8 39 5a f2 d2 41 19 33 8e 5b 05 5e 2f d1 ca 24 

        cipherSuite         TLS_RSA_WITH_AES_256_CBC_SHA

        compressionMethod                   NULL

      Certificate

      ServerHelloDone

3 3  0.0141 (0.0131)  C>S  Handshake

      ClientKeyExchange

3 4  0.0141 (0.0000)  C>S  ChangeCipherSpec

3 5  0.0141 (0.0000)  C>S  Handshake

3 6  0.0155 (0.0013)  S>C  Alert

    level           fatal

    value           handshake_failure

3    0.0155 (0.0000)  S>C  TCP FIN

3    0.0165 (0.0009)  C>S  TCP FIN

ssldump using Firefox 43 (Passes):

    New TCP connection 1: 192.168.100.125(55099) <-> 192.168.100.231(443)

    1 1  0.0007 (0.0007)  C>S  Handshake

  ClientHello

    Version 3.3 

    cipher suites

    TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

    TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

    TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

    TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA

    TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

    TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

    TLS_DHE_RSA_WITH_AES_128_CBC_SHA

    TLS_DHE_RSA_WITH_AES_256_CBC_SHA

    TLS_RSA_WITH_AES_128_CBC_SHA

    TLS_RSA_WITH_AES_256_CBC_SHA

    TLS_RSA_WITH_3DES_EDE_CBC_SHA

    compression methods

              NULL

    1 2  0.0012 (0.0004)  S>C  Handshake

  ServerHello

    Version 3.3 

    session_id[32]=

      0f 16 00 00 ec 24 3b 75 10 f0 53 c4 45 d3 df ef 

      97 91 f0 9a b8 fe c2 98 5d 15 fd 11 ed 2f 55 58 

    cipherSuite         TLS_RSA_WITH_AES_256_CBC_SHA

    compressionMethod                   NULL

  Certificate

  ServerHelloDone

    1 3  0.0031 (0.0018)  C>S  Handshake

  ClientKeyExchange

    1 4  0.0031 (0.0000)  C>S  ChangeCipherSpec

    1 5  0.0031 (0.0000)  C>S  Handshake

    1 6  0.0053 (0.0022)  S>C  ChangeCipherSpec

    1 7  0.0056 (0.0002)  S>C  Handshake

    1 8  0.2922 (0.2865)  C>S  application_data

    1 9  0.3330 (0.0408)  S>C  Handshake

    1 10 0.3337 (0.0006)  C>S  Handshake

    1 11 0.3368 (0.0031)  S>C  Handshake

    1 12 0.3473 (0.0104)  C>S  Handshake

    1 13 0.3473 (0.0000)  C>S  ChangeCipherSpec

    1 14 0.3473 (0.0000)  C>S  Handshake

    1 15 0.3500 (0.0026)  S>C  ChangeCipherSpec

    1 16 0.3501 (0.0001)  S>C  Handshake

    1 17 0.3512 (0.0011)  S>C  application_data

    1 18 0.3779 (0.0266)  C>S  application_data

    New TCP connection 2: 192.168.100.125(55102) <-> 192.168.100.231(443)

    2 1  0.0008 (0.0008)  C>S  Handshake

  ClientHello

    Version 3.3 

    resume [32]=

      b3 15 00 00 94 41 0f d7 0f ce 39 45 82 5e 53 85 

      b4 4f de 6d 1c f7 23 16 c6 8b bb d6 96 d9 53 c5 

    cipher suites

    TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

    TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

    TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

    TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA

    TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

    TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

    TLS_DHE_RSA_WITH_AES_128_CBC_SHA

    TLS_DHE_RSA_WITH_AES_256_CBC_SHA

    TLS_RSA_WITH_AES_128_CBC_SHA

    TLS_RSA_WITH_AES_256_CBC_SHA

    TLS_RSA_WITH_3DES_EDE_CBC_SHA

    compression methods

              NULL

    2 2  0.0011 (0.0003)  S>C  Handshake

  ServerHello

    Version 3.3 

    session_id[32]=

      b3 15 00 00 94 41 0f d7 0f ce 39 45 82 5e 53 85 

      b4 4f de 6d 1c f7 23 16 c6 8b bb d6 96 d9 53 c5 

    cipherSuite         TLS_RSA_WITH_AES_256_CBC_SHA

    compressionMethod                   NULL

    2 3  0.0012 (0.0000)  S>C  ChangeCipherSpec

    2 4  0.0018 (0.0006)  S>C  Handshake

    1 19 0.3804 (0.0025)  S>C  application_data

    2 5  0.0025 (0.0006)  C>S  ChangeCipherSpec

    2 6  0.0025 (0.0000)  C>S  Handshake

    2 7  0.0033 (0.0007)  C>S  application_data

    2 8  0.0057 (0.0023)  S>C  Handshake

    2 9  0.0062 (0.0005)  C>S  Handshake

    2 10 0.0072 (0.0010)  S>C  Handshake

    2 11 0.0210 (0.0137)  C>S  Handshake

    2 12 0.0210 (0.0000)  C>S  ChangeCipherSpec

    2 13 0.0210 (0.0000)  C>S  Handshake

    2 14 0.0246 (0.0035)  S>C  ChangeCipherSpec

    2 15 0.0246 (0.0000)  S>C  Handshake

    2 16 0.0250 (0.0003)  S>C  application_data

clien certificate authentication

Babak_AA_246963
Sep 23, 2016
I figured out the issue I was facing. It had to do with TLS Extended Master Secret and the BIG-IP was failing to decrypt the handshake. The extended master secret changes the way pre-master secret is generated for TLS sessions and I suspect BIG-IP fails to detect its presence and calculates the pre-master secret as if extended master secret is not in place, anyways I've written my experience in a couple of blog posts in case someones willing to get into the details: TLS Extended Master Secret Breaking SSL Proxies.

Solution

As for the solution, until BIG-IP adds this feature (decrypting sessions where extended master secret is used) I disabled it on my web server (The threat it was mitigating was minimal in my case when the choice is between having a WAF or having extended master secret enabled, it basically prevents rogue CAs to create bogus certificates and use them to MITM live TLS sessions, more details in the blog post).

Disabling TLS Extended Master Secret in Windows Server/IIS:

For IIS you'd have to go into registry and under SCHANNEL configurations add the following key:

Under HKLM\System\CurrentControlSet\Control\SecurityProviders\Schannel:

Add DisableServerExtendedMasterSecret as REG_DWORD with the value of 1 (anything other than 0 works)

The setting applies immediately and you don't need to restart the server.

14 Replies

Saravanan_M_K
Employee
Jul 06, 2016
Try setting the SSL logging level to "Debug" on the LTM and see what debug info it logs in the /var/log/ltm for the failure case. It may give more clue. After the testing, make sure to change the logging level back to the original value.

What is the version of IIS server/OS are you using?
Babak_AA_246963
Altostratus
Jul 06, 2016
ltm logs say:

srv warning tmm[12511]: 01260009:4: Connection error: hud_ssl_handler:1224: codec alert(20)

srv info tmm[12511]: 01260013:6: SSL Handshake failed for TCP 192.168.100.215:443 -> 192.168.100.230:51271

srv info tmm[12511]: 01260013:6: SSL Handshake failed for TCP 192.168.100.230:51271 -> 192.168.100.215:443

I'm using windows server 2012 and IIS 8.

Let me point out again that old web browsers and openssl cli connects through the LTM/ASM correctly but modern browsers fail to do so and get a handshake error. In order to isolate the issue I made sure all my tests are under same protocol/cipher suite (TLS_RSA_WITH_AES_256_CBC_SHA).

I looked at packet captures from BIG-IP's point of view and web servers point of view. The cases where there's a handshake failure alert at BIG-IP, the webserver sees a FIN followed by a RST after client sends change cipher spec and it's the BIG-IP resetting the connection due to handshake failure (I suspect it fails to decrypt the handshake for some reason). But in the case where openssl cli or older browsers are used the BIG-IP happily decrypts the handshake and the connection goes through it.
Hannes_Rapp
Nimbostratus
Jul 06, 2016
Check if your client-auth certificate, client-ssl certificate and all certificates in CA intermediary certificate bundle are signed with modern SHA algorithms, or if there are any certs signed by the deprecated SHA1. Note that the very root certificate of your CA may still remain SHA1-signed, but any intermediary certificates above it must be SHA256-signed (or stronger). Any of this does not apply to SSL handshakes and SHA in a cipher suite is still OK.

Newer browsers are becoming more and more restrictive with SHA1 certificates and this could be one potential cause to your issue. This OpenSSL command can help you get started:

openssl x509 -text -in '/Path/To/Cert.crt' | grep "Signature Algorithm" Signature Algorithm: sha256WithRSAEncryption
- Babak_AA_246963
  Altostratus
  Jul 07, 2016
  Apart from the root certificate which was SHA1, the rest of the certificates where using SHA256. but the fact that the BIG-IP sends a handshake failure and resets the connection has to do something with an issue on BIG-IP side rather than clients browser being unhappy, correct me if this assumption is incorrent.
- Hannes_Rapp
  Nimbostratus
  Jul 07, 2016
  You may be right, but we cannot really tell, as in your own words, there are browsers with which the whole solution works. To further isolate the issue, and eliminate variables, perhaps disabling Client-Auth for a test would be worth it. If the SSL handshake then completes successfully, it can at least be confirmed that it's a SSL-MA problem, and maybe tweaking some settings will do the trick. After all, it's also possible you're looking to solve a problem caused by a bug, so I recommend to open a support case with F5.
Hannes_Rapp_162
Nacreous
Jul 06, 2016
Check if your client-auth certificate, client-ssl certificate and all certificates in CA intermediary certificate bundle are signed with modern SHA algorithms, or if there are any certs signed by the deprecated SHA1. Note that the very root certificate of your CA may still remain SHA1-signed, but any intermediary certificates above it must be SHA256-signed (or stronger). Any of this does not apply to SSL handshakes and SHA in a cipher suite is still OK.

Newer browsers are becoming more and more restrictive with SHA1 certificates and this could be one potential cause to your issue. This OpenSSL command can help you get started:

openssl x509 -text -in '/Path/To/Cert.crt' | grep "Signature Algorithm" Signature Algorithm: sha256WithRSAEncryption
- Babak_AA_246963
  Altostratus
  Jul 07, 2016
  Apart from the root certificate which was SHA1, the rest of the certificates where using SHA256. but the fact that the BIG-IP sends a handshake failure and resets the connection has to do something with an issue on BIG-IP side rather than clients browser being unhappy, correct me if this assumption is incorrent.
- Hannes_Rapp_162
  Nacreous
  Jul 07, 2016
  You may be right, but we cannot really tell, as in your own words, there are browsers with which the whole solution works. To further isolate the issue, and eliminate variables, perhaps disabling Client-Auth for a test would be worth it. If the SSL handshake then completes successfully, it can at least be confirmed that it's a SSL-MA problem, and maybe tweaking some settings will do the trick. After all, it's also possible you're looking to solve a problem caused by a bug, so I recommend to open a support case with F5.
JG
Cumulonimbus
Jul 07, 2016
You can remove ECDHE-RSA-AES256-CBC-SHA from F5's cipher suite in your client-side SSL profile and see if that makes a difference.
Mickael_Srutows
Nimbostratus
Sep 23, 2016
Hello,

I have the same problem, you have a solution or not ? I have this problem since few days.

Thanks.
Babak_AA_246963
Altostratus
Sep 23, 2016
I figured out the issue I was facing. It had to do with TLS Extended Master Secret and the BIG-IP was failing to decrypt the handshake. The extended master secret changes the way pre-master secret is generated for TLS sessions and I suspect BIG-IP fails to detect its presence and calculates the pre-master secret as if extended master secret is not in place, anyways I've written my experience in a couple of blog posts in case someones willing to get into the details: TLS Extended Master Secret Breaking SSL Proxies.

Solution

As for the solution, until BIG-IP adds this feature (decrypting sessions where extended master secret is used) I disabled it on my web server (The threat it was mitigating was minimal in my case when the choice is between having a WAF or having extended master secret enabled, it basically prevents rogue CAs to create bogus certificates and use them to MITM live TLS sessions, more details in the blog post).

Disabling TLS Extended Master Secret in Windows Server/IIS:

For IIS you'd have to go into registry and under SCHANNEL configurations add the following key:

Under HKLM\System\CurrentControlSet\Control\SecurityProviders\Schannel:

Add DisableServerExtendedMasterSecret as REG_DWORD with the value of 1 (anything other than 0 works)

The setting applies immediately and you don't need to restart the server.
- mfkk531_168091
  Nimbostratus
  Aug 24, 2017
  I'm experiencing similar issue after upgrade to v12.1.2 Final.
  
  Any recommendation on how to fix this if the pool members are Apache servers ?
  
  Thanks!
Troy_Murray_196
Cirrus
Jan 16, 2017
AskF5 Support Article "K34019109: SSL handshakes using TLS extended master secret extensions fail when Proxy SSL is enabled" with this information can be found at https://support.f5.com/csp/article/K34019109
ML
Nimbostratus
Jan 03, 2018
No more errors in logs with 12.1.3.