F5 Server SSL Profile using TLS 1.0 instead of TLS 1.2

Well the pcap's if you've shared, i've seen this behavior too. Sometimes the IIS server schannel fails to reply back with the fatal alert message. It simply resets the the connection.

This puts hard to know what is causing the failure of SSL. There are so many SSL alert codes, which often gives us clue to where to look at for resolution, like protocol mismatch, cipher mismatch etc.

Regarding the version header protocols, Rodrigo has answered it clearly. As per the RFC, the lowest version has to be sent across in the record layer and the highest supported protocol in the handshake protocol.

Yes there have been cases where the record layers protocol version have caused issues too, but in those scenarios its the mistake of the server for being not compliant with the RFC standards.

As part of troubleshooting, can you please confirm if its a protocol issue or a cipher issue, sometime SHA1 could be blocked on the server side too.

Also you should think about the ssl session ID too, sometimes when the GTM/LTM sends health monitor probes, the session id gets stuck too.

To note, this SSL session ID gets cleared only when their is a proper reset packet coming back. But in schannel, the alert protocol never comes and this makes it hard for the LTM/GTM to understand clear it from its memory. But this is for health monitor. Lets shift gear to serverssl profile.

Long story short: I experienced this behavior too. I had to 1st check whether its a protocol problem or the SHA1 or some ciphers. Use openssl to confirm the same.

#To test tls1.0
openssl s_client -connect IP:PORT -tls1
 
#To test tls1.1
openssl s_client -connect IP:PORT -tls1_1
 
#To test tls1.2
openssl s_client -connect IP:PORT -tls1_2
 
#To test SHA1
openssl s_client -cipher 'SHA' -connect IP:PORT -tls1_2
 
#To test SHA2
openssl s_client -cipher 'SHA256' -connect IP:PORT -tls1_2

Once you have the above data, you can come to conclusion what could be the problem. If you suspect in the Client Hello, only a certain set of ciphers are being sent and your server is limited to a particular set of ciphers, you can check that too using ssldump.

Take a normal pcap and run ssldump to it.

Regarding the fix, once you find if its TLS 1.1 or SHA, try one after one and see the results,

#To block the weak ciphers
tmsh modify ltm profile client-ssl <client-ssl-profile> ciphers "!3DES:!DH:!ADH:!EDH:!MD5:!EXPORT:!DES:!RC4:!SSLv3:!TLSv1:!TLSv1_1:!SHA1"
 
#To block the old protocols
tmsh modify ltm profile client-ssl <client-ssl-profile> options { dont-insert-empty-fragments no-sslv2 no-sslv3 no-tlsv1 no-tlsv1.1 }

Keep us posted.