F5 Server SSL Profile using TLS 1.0 instead of TLS 1.2

[pstavr]

Hi

I have an F5 virtual server that does SSL inspection so it has a client ssl profile and a server ssl profile. The backend server is running on a Windows Server 2019 / IIS and it only accepts TLS 1.1 and 1.2 clients. Since the F5 acts as a client in this case towards the Windows Server 2019, I have created a server ssl profile which forces the F5 to use TLS 1.2 only (SSL Proxy is disabled). My problem is that during the Client Hello from the F5 towards the Windows Server, TLS 1.0 is used. So the backend server immediatelly sends a RST ACK without sending Server Hello for supported ciphers etc.

While doing a capture on Wireshark, I saw that TLS 1.0 is used. And further down in the same TCP packet it mentions TLS 1.2.

This is driving me crazy. Is there any way we can enforce F5 to use 1.2 only during Client Hello?

[pstavr]

Here is how the Wireshark looks. Notice the highlighted parts TLS 1.0 on top, but further down TLS 1.2

[pstavr]

And here is the Cipher string I am using on the server ssl profile.

F5 is running on 14.1.2.1

Any help / ideas would be greatly appreciated. Thanks!

[NUT2889]

Hi,

For the "Options" property. You can move "No TLSv1" from avaialbe list to enabled options. Then try again.

[pstavr]

Thank you for your reply. I have already done that. Same result. Initially I created the server ssl profile using "tmsh create /ltm profile server-ssl testv12 ciphers TLSv1_2"

I was getting the RST ACK due to TLSV1.

My next move was to create an SSL profile through the GUI and use the "Options" property to remove SSLv3, TLV1, TLSv1.1. That did not work either. Again, RST ACK since the Client Hello was TLS 1.0.

[Rodrigo_Albuque]

Hi pstavr,

BIG-IP is NOT using TLS 1.0.

Although not specified in RFC, the de facto standard used by all major browsers and TLS stacks is to set the ClientHello as follows:

SSL Record:
  Content Type: Handshake (22)
  Version: $LOWEST_VERSION <----
  Handshake Record:
    Handshake Type: Client Hello (1)
    Version: $HIGHEST_VERSION <----

The BIG-IP system implementation tells the TLS peer that the system supports only TLS versions from the $LOWEST_VERSION through the $HIGHEST_VERSION.

Back end server is then supposed to reply with the selected version correctly and move on.

If your server is failing because of this, then there's an issue with your server but definitely NOT BIG-IP.

Please, Is there a Fatal Alert sent by the server with the error code? You should look into that.

[pstavr]

Hi Rodrigo

Thank you for your reply. We are already trying a few tweaks on the Windows Server 2019 / IIS environment for fixing this. I understand your comment and it makes sense. What I do not understand however is why would F5 use TLS 1.0 initially to contact the backend server during the Client Hello. I am using a server SSL profile which limits the ciphers to TLS 1.2 only. I should also mention that Windows 10 clients accessing the backend server directly work perfectly fine. While doing a Wireshark capture on the Windows Clients, the Client Hello is entirely on TLS 1.2 (both lowest and highest version). Hence my questioning on why F5 would send a TLS 1.0 Client Hello. The packet is marked as TLS1 on Wireshark.

[Rodrigo_Albuque]

pstavr,

That is what I'm trying to explain.

The TLS1.0 Client Hello BIG-IP is sending is just a convention to signal which version it supports.

If your Windows client only sends TLS1.2/TLS1.2 it means it only supports TLS1.2 and nothing else.

Because BIG-IP supports TLS1.0, TLS1.1 and TLS1.2 it signals to server TLS1.0/TLS1.2.

This is expected behaviour.

And FYI, I took a capture on my Ubuntu box and this is my client hello:

Does it make sense?

[pstavr]

Sort of does. I guess I would say it makes sense if you use DEFAULT on the ciphers. But if you restrict it to TLS 1.2, why would the F5 send TLS 1.0 as the lowest? I do 100% agree though that there is something wrong with the backend server so we need to tweak it to be more "open", at least during the negotiation process.

[Rodrigo_Albuque]

If you're restricting BIG-IP to ONLY TLS1.2 then that might be a bug as BIG-IP is supposed to send TLS1.2/TLS1.2 in my understanding. However, the fact that BIG-IP signals IIS that it supports TLS1.0 to TLS1.2 is not supposed to trigger connection failure. The fact that BIG-IP also supports TLS1.0 and TLS1.1 is not supposed to break the connection. IIS is supposed to reply with TLS1.2 version and keep going. I'd say this is a broken behaviour from IIS and I believe the easiest thing to solve your issue is just to have a quick look at IIS log when the connection fails. Other than that, you can definitely open a low-priority (Sev4) Support ticket with F5 and ask them to file a low-priority bug for this behaviour.

[Simon_Blakely]

> But if you restrict it to TLS 1.2, why would the F5 send TLS 1.0 as the lowest?

Late to the party, but you might appreciate this explanation:

F5 SSL profiles distinguish between Ciphers and Protocols.

Your cipher string restricts the cipher list to only those ciphers that are supported on TLSv1.2, but the server-ssl profile still supports the TLS Protocols from TLSv1 - TLSv1.2.

You can only negotiate TLSv1.2 ciphers on the TLSv1.2 Protocol, but you haven't explicitly instructed the profile to disable the earlier TLS Protocols, so it offers them.

You can disable TLS Protocols using the Options settings (No TLSv1, No TLSv1.1 ).

I hope this provides some useful information.

[pstavr]

If you see the yellow-highlighted TLS version, that is F5 sending Client Hello. Its TLSv1. The exact same PCAP capture on Windows 10 clients accessing the Windows Server 2019 directly shows up TLSv1.2.

[Rodrigo_Albuque]

Which version of BIG-IP are you using?

[pstavr]

14.1.2.1

[Rodrigo_Albuque]

I'm not convinced this is the problem.

Are you able to access IIS logs to see what it says about this handshake?

It could be cipher mismatch, for example.

[pstavr]

We extracted a list of the ciphers supported from IIS. Quite a few were common with F5 so they were included in the Client Hello. Thats why I am saying I agree with you that it should accept one of them and proceed further by sending a Server Hello.

I was thinking of creating a list of ciphers based on the extracted ones from IIS and create a custom cipher list to be 100% match for the one on IIS. But even if that would work, it still does not explain why a RST ACK is sent now when there are so many common ciphers between the two...

[nick]

Is it a .NET application by any chance?

[pstavr]

I believe so yes. I have to check that with the app team, but I think its a .NET application hosted on that IIS.

[nick]

I had this issue and had to add a registry key:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319]

"SystemDefaultTlsVersions"=dword:00000001

"SchUseStrongCrypto"=dword:00000001

https://docs.microsoft.com/en-us/dotnet/framework/network-programming/tls

[pstavr]

Thank you. I ll let them know. I believe they did modify a GPO object but maybe that was IIS related. I try everything out on Monday and come back with feedback. Thank you all!

[NUT2889]

Hi,

You can use this tool help identify the TLS version and Cipher suite that your IIS was enabled.

https://www.bolet.org/TestSSLServer/

Example is over here.

[jaikumar_f5]

Well the pcap's if you've shared, i've seen this behavior too. Sometimes the IIS server schannel fails to reply back with the fatal alert message. It simply resets the the connection.

This puts hard to know what is causing the failure of SSL. There are so many SSL alert codes, which often gives us clue to where to look at for resolution, like protocol mismatch, cipher mismatch etc.

Regarding the version header protocols, Rodrigo has answered it clearly. As per the RFC, the lowest version has to be sent across in the record layer and the highest supported protocol in the handshake protocol.

Yes there have been cases where the record layers protocol version have caused issues too, but in those scenarios its the mistake of the server for being not compliant with the RFC standards.

As part of troubleshooting, can you please confirm if its a protocol issue or a cipher issue, sometime SHA1 could be blocked on the server side too.

Also you should think about the ssl session ID too, sometimes when the GTM/LTM sends health monitor probes, the session id gets stuck too.

To note, this SSL session ID gets cleared only when their is a proper reset packet coming back. But in schannel, the alert protocol never comes and this makes it hard for the LTM/GTM to understand clear it from its memory. But this is for health monitor. Lets shift gear to serverssl profile.

Long story short: I experienced this behavior too. I had to 1st check whether its a protocol problem or the SHA1 or some ciphers. Use openssl to confirm the same.

#To test tls1.0
openssl s_client -connect IP:PORT -tls1
 
#To test tls1.1
openssl s_client -connect IP:PORT -tls1_1
 
#To test tls1.2
openssl s_client -connect IP:PORT -tls1_2
 
#To test SHA1
openssl s_client -cipher 'SHA' -connect IP:PORT -tls1_2
 
#To test SHA2
openssl s_client -cipher 'SHA256' -connect IP:PORT -tls1_2

Once you have the above data, you can come to conclusion what could be the problem. If you suspect in the Client Hello, only a certain set of ciphers are being sent and your server is limited to a particular set of ciphers, you can check that too using ssldump.

Take a normal pcap and run ssldump to it.

Regarding the fix, once you find if its TLS 1.1 or SHA, try one after one and see the results,

#To block the weak ciphers
tmsh modify ltm profile client-ssl <client-ssl-profile> ciphers "!3DES:!DH:!ADH:!EDH:!MD5:!EXPORT:!DES:!RC4:!SSLv3:!TLSv1:!TLSv1_1:!SHA1"
 
#To block the old protocols
tmsh modify ltm profile client-ssl <client-ssl-profile> options { dont-insert-empty-fragments no-sslv2 no-sslv3 no-tlsv1 no-tlsv1.1 }

Keep us posted.

[pstavr]

Hi!

All tests failed today. But I have some extra clues.

1. The app running on the background is a .NET application
2. The registry tweaks for the .NET app basically failed. Same issue.
3. We also turned off the app its self, and tried to use IIS default pages, just to see if the communication between F5 and the backend server would be OK. And it is, it works perfectly fine. So its the ,NET app that breaks it.

The weird part is that without modifying anything on F5, it works with simple IIS on the backend, but it does not with the .NET app enabled. The only difference I can see in WireShark is that the successful Client Hello done from the F5 wowards the backend server, is done using TLS 1.2. Where one that fails uses TLSv1 instead for the Client Hello. Both mark 1.0 as the minimum and 1.2 as the maximum accewpted TLS though.

Have a look at the successful attempts against IIS, and compare them to the unsuccesful against the .NET app. Only difference is that the version used for the Client Hello part. Once again, thank you all for your ideas and feedback.

[jaikumar_f5]

Lets not stick to the record layer that it could be the issue, we should check on the other angles too (ciphers, sha, protocols).... Can you share your pcaps of failed and working one. Expand the client hello packet. Also what is there in the handshake protocol version is what would be shown in the pcap protocol info.

Also share the test results of below,

#To test tls1.1
openssl s_client -connect IP:PORT -tls1_1
 
#To test tls1.2
openssl s_client -connect IP:PORT -tls1_2
 
#To test SHA1
openssl s_client -cipher 'SHA' -connect IP:PORT -tls1_2
 
#To test SHA2
openssl s_client -cipher 'SHA256' -connect IP:PORT -tls1_2

[pstavr · Accepted solution]

Hi all.

I found the root cause. The problem was related to the .NET app using SNI. By default the F5 doesn't do that.

https://devcentral.f5.com/s/articles/ssl-profiles-part-7-server-name-indication

So basically I just followed the fix in the above article, I defined a server name and the backend service started sending Server Hello etc. Everything works fine now!

Thank you all for your responses, as quite a few of them were helpful on identifying that the issue is with the app, and I could also spot a few things that were not proper on the negotiation part.

[jaikumar_f5]

Interesting, Glad your issue is resolved. One thing I learnt over the years is to look around for all possibilities. Never stick to a point and think it could be the problem.​ Mark your own answer as solution provided and close the thread.

[NUT2889]

Hi,

I have few experience and share with you.

Some of administrator has enabled SNI on web server e.g. IIS, Nginx, Apache. But him didn't tell me to know.

When we configure F5 to monitor via HTTP or HTTPs. The result is pool offline. Sometimes we need to re-check administrator for this setting to reduce ping-pong situation.

[jaikumar_f5]

Actually, it's quite common, in the monitor the Host header section matters a lot.

Troubleshooting health check is one thing, and troubleshooting application traffic flow is another thing.​

[NUT2889]

Hi,

Thank you for your feedback.

From my perspective "Host header" and "SNI" is different level.

• Host header is in web application server level.
• SNI is in SSL/TLS protocol level.

[Bobow]

Hi pstavr,

Is this issue resolved? i have the same problem with this.

What solution do you use to improve from the F5 side?

Thanks

[pstavr]

Hi Bonow.

Yes, it was happening because the backend servers required SNI support. I have posted an article above from F5 where you can see the solution.

[Bobow]

Well noted. Thank You Bro!

[jaikumar_f5]

 , did your issue resolve. The reason I ask is, there are multiple scenarios where this could happen. One such is shared above, its not that yours could also be as same as above. If its same and issue is fixed. Good to know...