serverssl behavior in 11.4.x

Started to answer this one, but realized that none of them was viable. The client suggests certificates, but the server chooses. That would not explain why it would work if you REMOVED a cipher from the list though. Out of curiosity, do you see any entries in the application logs? /Patrik

And can you match the client hello list with the server list? :)

What is different about the new server certificate? ECDSA vs RSA? It almost sounds like your server is requiring an unsupported cipher suite when using TLS1.2 and can happily negotiate something supported if using a lower protocol. I'd highly recommend doing an SSLDUMP capture on the server side to see what's happening.

ssldump -AdNn -i [0.0 or internal VLAN name] port 443 [and any other display filters]

You're looking for when the error is thrown and what was sent immediately prior. For example, if the server sends an alert immediately after the F5's ClientHello, then you can suspect the client didn't present a protocol and cipher suite combination that the server would accept. What then changes when you disable TLS1.2?

The Deployment Guide for Exchange 2013 contains the following interesting note:

SSL ciphers used in the Server SSL profile in 11.4.x are not compatible with those in some versions of Microsoft Internet Information Server (IIS).

This seems to be a bug in only this version.

And yes I also made a ssldump, but didn't have it available anymore (and can't reproduce it again). But I can remember that after the client hello the server immediately sends a RST (not a FIN). Right now without TLS1.2 it looks like this:

New TCP connection 1: 10.71.65.97(43749) <-> 10.76.144.52(443)
1 1  0.0003 (0.0003)  C>SV3.2(107)  Handshake
  ClientHello
    Version 3.2 
    random[32]=
      22 62 5d 20 0e 4d dd 89 b9 d0 25 d5 c1 fc cb 63 
      62 d8 f7 ac 8f a6 bd ea d8 a0 25 85 70 6d a7 8d 
    resume [32]=
      99 33 00 00 3d 97 b1 db b9 89 e9 b0 25 7f 8e aa 
      51 35 35 d3 08 63 47 2a 04 3e 64 b1 21 d1 4a 81 
    cipher suites
    TLS_RSA_WITH_RC4_128_SHA
    TLS_RSA_WITH_AES_128_CBC_SHA
    TLS_RSA_WITH_AES_256_CBC_SHA
    TLS_RSA_WITH_3DES_EDE_CBC_SHA
    Unknown value 0xc013
    Unknown value 0xc014
    Unknown value 0xc012
    Unknown value 0xff
    compression methods
              NULL
1 2  0.0009 (0.0005)  S>CV3.2(81)  Handshake
  ServerHello
    Version 3.2 
    random[32]=
      56 0c df 67 99 9e 1b 83 f9 77 e2 27 24 8e 13 c1 
      b0 a2 f1 7b 5b 65 88 fe e1 91 95 64 88 a1 08 c7 
    session_id[32]=
      99 33 00 00 3d 97 b1 db b9 89 e9 b0 25 7f 8e aa 
      51 35 35 d3 08 63 47 2a 04 3e 64 b1 21 d1 4a 81 
    cipherSuite         Unknown value 0xc014
    compressionMethod                   NULL

If I check the above I also can remember that with TLS 1.2 there was not this "resume [32]=" line included.

Here is the sniffer output from the failure:

Ciao Stefan 🙂

The resume field just means the client is attempting to resume a previous session, so you're not technically looking at an initial SSL handshake. Also notice that your client is presenting a bunch of elliptic curve ciphers with TLS1.2 that it doesn't for TLS1.1 and below. It's hard to say exactly what the issue is since the server is the one throwing the error, but since it's happening immediately after the clienthello, it's a good bet the server doesn't like some cipher or extension that the client is sending.

There's a few ways to troubleshoot this. First, here's a really cool Bash script you can point at your server to see what ciphers and protocols it actually supports:

http://www.tuxad.de/scripts/ssltest.sh

That may immediately show you the problem. Otherwise you can start testing different protocol and cipher options with the openssl s_client command.

openssl s_client -connect 192.168.10.10:443

There's a "-cipher" option to this command that allows you to tweak the ciphers that you send, so for example:

openssl s_client -connect 192.168.10.10:443 -cipher 'ECDHE-RSA-AES128-SHA'

You can then run through all of the ciphers that you saw in the ClientHello of the failed request, or the ciphers listed in the Bash script.