BIG-IP Proxy SSL 12.1 Handshake Failure
I set up SSL Proxy in order to do client certificate authentication on my IIS web server on LTM 12.1 firmware. The setup is working fine on Firefox version 43, IE 10 and OpenSSL but it fails on Chrome 51, Firefox 47 and IE 11.
I've captured the packets. Clients use TLS1 or TLS1.2 using the same ciphersuite of TLS_RSA_WITH_AES_256_CBC_SHA (0x0035), the same process takes place for the passing and failing cases.
- Client Hello
- Server Hello, Certificate, Server Hello Done
- Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message 4. 4.1 Either Server sends Change Cipher Spec and then Application Data gets transfered Or 4.2 The server sends Alert level: Fatal, Descrition: Handshake Failure
So I suspect the BIG-IP fails to decrypt the handshake sent by the client in some cases but I can't figure out why because there's nothing different between failing and passing tests.
ssldump using Firefox 47 (Fails):
New TCP connection 1: 192.168.100.125(55041) <-> 192.168.100.231(443)
1 1 0.0027 (0.0027) C>S Handshake
ClientHello
Version 3.3
cipher suites
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
Unknown value 0xcca9
Unknown value 0xcca8
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_3DES_EDE_CBC_SHA
compression methods
NULL
1 2 0.0033 (0.0005) S>C Handshake
ServerHello
Version 3.3
session_id[32]=
d9 0a 00 00 3e 11 22 ac e2 c2 00 f5 9a 41 35 53
43 6a 9e a5 e0 26 32 e4 f8 38 2e ca 72 3c fb 93
cipherSuite TLS_RSA_WITH_AES_256_CBC_SHA
compressionMethod NULL
Certificate
ServerHelloDone
1 3 0.0185 (0.0151) C>S Handshake
ClientKeyExchange
1 4 0.0185 (0.0000) C>S ChangeCipherSpec
1 5 0.0185 (0.0000) C>S Handshake
1 6 0.0196 (0.0011) S>C Alert
level fatal
value handshake_failure
1 0.0197 (0.0000) S>C TCP FIN
1 0.0205 (0.0008) C>S TCP FIN
New TCP connection 2: 192.168.100.125(55042) <-> 192.168.100.231(443)
2 1 0.0005 (0.0005) C>S Handshake
ClientHello
Version 3.2
cipher suites
Unknown value 0x5600
Unknown value 0xcca9
Unknown value 0xcca8
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_3DES_EDE_CBC_SHA
compression methods
NULL
2 2 0.0010 (0.0005) S>C Handshake
ServerHello
Version 3.2
session_id[32]=
85 48 00 00 8f 2a ae 80 b8 d7 e9 e2 47 c0 15 4e
e8 af 69 6f 2d b9 b8 d6 ed d5 29 3c a3 a3 44 b3
cipherSuite TLS_RSA_WITH_AES_256_CBC_SHA
compressionMethod NULL
Certificate
ServerHelloDone
2 3 0.0145 (0.0134) C>S Handshake
ClientKeyExchange
2 4 0.0145 (0.0000) C>S ChangeCipherSpec
2 5 0.0145 (0.0000) C>S Handshake
2 6 0.0158 (0.0013) S>C Alert
level fatal
value handshake_failure
2 0.0158 (0.0000) S>C TCP FIN
2 0.0162 (0.0003) C>S TCP FIN
New TCP connection 3: 192.168.100.125(55043) <-> 192.168.100.231(443)
3 1 0.0005 (0.0005) C>S Handshake
ClientHello
Version 3.1
cipher suites
Unknown value 0x5600
Unknown value 0xcca9
Unknown value 0xcca8
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_3DES_EDE_CBC_SHA
compression methods
NULL
3 2 0.0010 (0.0004) S>C Handshake
ServerHello
Version 3.1
session_id[32]=
aa 41 00 00 04 82 07 3f ed 35 96 49 e2 c5 ba 79
f8 39 5a f2 d2 41 19 33 8e 5b 05 5e 2f d1 ca 24
cipherSuite TLS_RSA_WITH_AES_256_CBC_SHA
compressionMethod NULL
Certificate
ServerHelloDone
3 3 0.0141 (0.0131) C>S Handshake
ClientKeyExchange
3 4 0.0141 (0.0000) C>S ChangeCipherSpec
3 5 0.0141 (0.0000) C>S Handshake
3 6 0.0155 (0.0013) S>C Alert
level fatal
value handshake_failure
3 0.0155 (0.0000) S>C TCP FIN
3 0.0165 (0.0009) C>S TCP FIN
ssldump using Firefox 43 (Passes):
New TCP connection 1: 192.168.100.125(55099) <-> 192.168.100.231(443)
1 1 0.0007 (0.0007) C>S Handshake
ClientHello
Version 3.3
cipher suites
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_3DES_EDE_CBC_SHA
compression methods
NULL
1 2 0.0012 (0.0004) S>C Handshake
ServerHello
Version 3.3
session_id[32]=
0f 16 00 00 ec 24 3b 75 10 f0 53 c4 45 d3 df ef
97 91 f0 9a b8 fe c2 98 5d 15 fd 11 ed 2f 55 58
cipherSuite TLS_RSA_WITH_AES_256_CBC_SHA
compressionMethod NULL
Certificate
ServerHelloDone
1 3 0.0031 (0.0018) C>S Handshake
ClientKeyExchange
1 4 0.0031 (0.0000) C>S ChangeCipherSpec
1 5 0.0031 (0.0000) C>S Handshake
1 6 0.0053 (0.0022) S>C ChangeCipherSpec
1 7 0.0056 (0.0002) S>C Handshake
1 8 0.2922 (0.2865) C>S application_data
1 9 0.3330 (0.0408) S>C Handshake
1 10 0.3337 (0.0006) C>S Handshake
1 11 0.3368 (0.0031) S>C Handshake
1 12 0.3473 (0.0104) C>S Handshake
1 13 0.3473 (0.0000) C>S ChangeCipherSpec
1 14 0.3473 (0.0000) C>S Handshake
1 15 0.3500 (0.0026) S>C ChangeCipherSpec
1 16 0.3501 (0.0001) S>C Handshake
1 17 0.3512 (0.0011) S>C application_data
1 18 0.3779 (0.0266) C>S application_data
New TCP connection 2: 192.168.100.125(55102) <-> 192.168.100.231(443)
2 1 0.0008 (0.0008) C>S Handshake
ClientHello
Version 3.3
resume [32]=
b3 15 00 00 94 41 0f d7 0f ce 39 45 82 5e 53 85
b4 4f de 6d 1c f7 23 16 c6 8b bb d6 96 d9 53 c5
cipher suites
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_3DES_EDE_CBC_SHA
compression methods
NULL
2 2 0.0011 (0.0003) S>C Handshake
ServerHello
Version 3.3
session_id[32]=
b3 15 00 00 94 41 0f d7 0f ce 39 45 82 5e 53 85
b4 4f de 6d 1c f7 23 16 c6 8b bb d6 96 d9 53 c5
cipherSuite TLS_RSA_WITH_AES_256_CBC_SHA
compressionMethod NULL
2 3 0.0012 (0.0000) S>C ChangeCipherSpec
2 4 0.0018 (0.0006) S>C Handshake
1 19 0.3804 (0.0025) S>C application_data
2 5 0.0025 (0.0006) C>S ChangeCipherSpec
2 6 0.0025 (0.0000) C>S Handshake
2 7 0.0033 (0.0007) C>S application_data
2 8 0.0057 (0.0023) S>C Handshake
2 9 0.0062 (0.0005) C>S Handshake
2 10 0.0072 (0.0010) S>C Handshake
2 11 0.0210 (0.0137) C>S Handshake
2 12 0.0210 (0.0000) C>S ChangeCipherSpec
2 13 0.0210 (0.0000) C>S Handshake
2 14 0.0246 (0.0035) S>C ChangeCipherSpec
2 15 0.0246 (0.0000) S>C Handshake
2 16 0.0250 (0.0003) S>C application_data
I figured out the issue I was facing. It had to do with TLS Extended Master Secret and the BIG-IP was failing to decrypt the handshake. The extended master secret changes the way pre-master secret is generated for TLS sessions and I suspect BIG-IP fails to detect its presence and calculates the pre-master secret as if extended master secret is not in place, anyways I've written my experience in a couple of blog posts in case someones willing to get into the details: TLS Extended Master Secret Breaking SSL Proxies.
Solution
As for the solution, until BIG-IP adds this feature (decrypting sessions where extended master secret is used) I disabled it on my web server (The threat it was mitigating was minimal in my case when the choice is between having a WAF or having extended master secret enabled, it basically prevents rogue CAs to create bogus certificates and use them to MITM live TLS sessions, more details in the blog post).
Disabling TLS Extended Master Secret in Windows Server/IIS:
For IIS you'd have to go into registry and under SCHANNEL configurations add the following key:
Under HKLM\System\CurrentControlSet\Control\SecurityProviders\Schannel: