Forum Discussion
BIG-IP Proxy SSL 12.1 Handshake Failure
- Sep 23, 2016
I figured out the issue I was facing. It had to do with TLS Extended Master Secret and the BIG-IP was failing to decrypt the handshake. The extended master secret changes the way pre-master secret is generated for TLS sessions and I suspect BIG-IP fails to detect its presence and calculates the pre-master secret as if extended master secret is not in place, anyways I've written my experience in a couple of blog posts in case someones willing to get into the details: TLS Extended Master Secret Breaking SSL Proxies.
Solution
As for the solution, until BIG-IP adds this feature (decrypting sessions where extended master secret is used) I disabled it on my web server (The threat it was mitigating was minimal in my case when the choice is between having a WAF or having extended master secret enabled, it basically prevents rogue CAs to create bogus certificates and use them to MITM live TLS sessions, more details in the blog post).
Disabling TLS Extended Master Secret in Windows Server/IIS:
For IIS you'd have to go into registry and under SCHANNEL configurations add the following key:
Under HKLM\System\CurrentControlSet\Control\SecurityProviders\Schannel:
ltm logs say:
srv warning tmm[12511]: 01260009:4: Connection error: hud_ssl_handler:1224: codec alert(20)
srv info tmm[12511]: 01260013:6: SSL Handshake failed for TCP 192.168.100.215:443 -> 192.168.100.230:51271
srv info tmm[12511]: 01260013:6: SSL Handshake failed for TCP 192.168.100.230:51271 -> 192.168.100.215:443
I'm using windows server 2012 and IIS 8.
Let me point out again that old web browsers and openssl cli connects through the LTM/ASM correctly but modern browsers fail to do so and get a handshake error. In order to isolate the issue I made sure all my tests are under same protocol/cipher suite (TLS_RSA_WITH_AES_256_CBC_SHA).
I looked at packet captures from BIG-IP's point of view and web servers point of view. The cases where there's a handshake failure alert at BIG-IP, the webserver sees a FIN followed by a RST after client sends change cipher spec and it's the BIG-IP resetting the connection due to handshake failure (I suspect it fails to decrypt the handshake for some reason). But in the case where openssl cli or older browsers are used the BIG-IP happily decrypts the handshake and the connection goes through it.
Recent Discussions
Related Content
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com