Forum Discussion
F5 Server SSL Profile using TLS 1.0 instead of TLS 1.2
Hi
I have an F5 virtual server that does SSL inspection so it has a client ssl profile and a server ssl profile. The backend server is running on a Windows Server 2019 / IIS and it only accepts TLS 1.1 and 1.2 clients. Since the F5 acts as a client in this case towards the Windows Server 2019, I have created a server ssl profile which forces the F5 to use TLS 1.2 only (SSL Proxy is disabled). My problem is that during the Client Hello from the F5 towards the Windows Server, TLS 1.0 is used. So the backend server immediatelly sends a RST ACK without sending Server Hello for supported ciphers etc.
While doing a capture on Wireshark, I saw that TLS 1.0 is used. And further down in the same TCP packet it mentions TLS 1.2.
This is driving me crazy. Is there any way we can enforce F5 to use 1.2 only during Client Hello?
Hi all.
I found the root cause. The problem was related to the .NET app using SNI. By default the F5 doesn't do that.
https://devcentral.f5.com/s/articles/ssl-profiles-part-7-server-name-indication
So basically I just followed the fix in the above article, I defined a server name and the backend service started sending Server Hello etc. Everything works fine now!
Thank you all for your responses, as quite a few of them were helpful on identifying that the issue is with the app, and I could also spot a few things that were not proper on the negotiation part.
- NUT2889Cirrostratus
Hi,
For the "Options" property. You can move "No TLSv1" from avaialbe list to enabled options. Then try again.
- pstavrCirrus
Thank you for your reply. I have already done that. Same result. Initially I created the server ssl profile using "tmsh create /ltm profile server-ssl testv12 ciphers TLSv1_2"
I was getting the RST ACK due to TLSV1.
My next move was to create an SSL profile through the GUI and use the "Options" property to remove SSLv3, TLV1, TLSv1.1. That did not work either. Again, RST ACK since the Client Hello was TLS 1.0.
Hi pstavr,
BIG-IP is NOT using TLS 1.0.
Although not specified in RFC, the de facto standard used by all major browsers and TLS stacks is to set the ClientHello as follows:
SSL Record: Content Type: Handshake (22) Version: $LOWEST_VERSION <---- Handshake Record: Handshake Type: Client Hello (1) Version: $HIGHEST_VERSION <----
The BIG-IP system implementation tells the TLS peer that the system supports only TLS versions from the $LOWEST_VERSION through the $HIGHEST_VERSION.
Back end server is then supposed to reply with the selected version correctly and move on.
If your server is failing because of this, then there's an issue with your server but definitely NOT BIG-IP.
Please, Is there a Fatal Alert sent by the server with the error code? You should look into that.
- pstavrCirrus
Here is how the Wireshark looks. Notice the highlighted parts TLS 1.0 on top, but further down TLS 1.2
- pstavrCirrus
And here is the Cipher string I am using on the server ssl profile.
F5 is running on 14.1.2.1
Any help / ideas would be greatly appreciated. Thanks!
- pstavrCirrus
Hi Rodrigo
Thank you for your reply. We are already trying a few tweaks on the Windows Server 2019 / IIS environment for fixing this. I understand your comment and it makes sense. What I do not understand however is why would F5 use TLS 1.0 initially to contact the backend server during the Client Hello. I am using a server SSL profile which limits the ciphers to TLS 1.2 only. I should also mention that Windows 10 clients accessing the backend server directly work perfectly fine. While doing a Wireshark capture on the Windows Clients, the Client Hello is entirely on TLS 1.2 (both lowest and highest version). Hence my questioning on why F5 would send a TLS 1.0 Client Hello. The packet is marked as TLS1 on Wireshark.
pstavr,
That is what I'm trying to explain.
The TLS1.0 Client Hello BIG-IP is sending is just a convention to signal which version it supports.
If your Windows client only sends TLS1.2/TLS1.2 it means it only supports TLS1.2 and nothing else.
Because BIG-IP supports TLS1.0, TLS1.1 and TLS1.2 it signals to server TLS1.0/TLS1.2.
This is expected behaviour.
And FYI, I took a capture on my Ubuntu box and this is my client hello:
Does it make sense?
- pstavrCirrus
Sort of does. I guess I would say it makes sense if you use DEFAULT on the ciphers. But if you restrict it to TLS 1.2, why would the F5 send TLS 1.0 as the lowest? I do 100% agree though that there is something wrong with the backend server so we need to tweak it to be more "open", at least during the negotiation process.
- pstavrCirrus
If you see the yellow-highlighted TLS version, that is F5 sending Client Hello. Its TLSv1. The exact same PCAP capture on Windows 10 clients accessing the Windows Server 2019 directly shows up TLSv1.2.
Which version of BIG-IP are you using?
- pstavrCirrus
14.1.2.1
- nickNimbostratus
Is it a .NET application by any chance?
- pstavrCirrus
I believe so yes. I have to check that with the app team, but I think its a .NET application hosted on that IIS.
- nickNimbostratus
I had this issue and had to add a registry key:
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319]
"SystemDefaultTlsVersions"=dword:00000001
"SchUseStrongCrypto"=dword:00000001
https://docs.microsoft.com/en-us/dotnet/framework/network-programming/tls
Well the pcap's if you've shared, i've seen this behavior too. Sometimes the IIS server schannel fails to reply back with the fatal alert message. It simply resets the the connection.
This puts hard to know what is causing the failure of SSL. There are so many SSL alert codes, which often gives us clue to where to look at for resolution, like protocol mismatch, cipher mismatch etc.
Regarding the version header protocols, Rodrigo has answered it clearly. As per the RFC, the lowest version has to be sent across in the record layer and the highest supported protocol in the handshake protocol.
Yes there have been cases where the record layers protocol version have caused issues too, but in those scenarios its the mistake of the server for being not compliant with the RFC standards.
As part of troubleshooting, can you please confirm if its a protocol issue or a cipher issue, sometime SHA1 could be blocked on the server side too.
Also you should think about the ssl session ID too, sometimes when the GTM/LTM sends health monitor probes, the session id gets stuck too.
To note, this SSL session ID gets cleared only when their is a proper reset packet coming back. But in schannel, the alert protocol never comes and this makes it hard for the LTM/GTM to understand clear it from its memory. But this is for health monitor. Lets shift gear to serverssl profile.
Long story short: I experienced this behavior too. I had to 1st check whether its a protocol problem or the SHA1 or some ciphers. Use openssl to confirm the same.
#To test tls1.0 openssl s_client -connect IP:PORT -tls1 #To test tls1.1 openssl s_client -connect IP:PORT -tls1_1 #To test tls1.2 openssl s_client -connect IP:PORT -tls1_2 #To test SHA1 openssl s_client -cipher 'SHA' -connect IP:PORT -tls1_2 #To test SHA2 openssl s_client -cipher 'SHA256' -connect IP:PORT -tls1_2
Once you have the above data, you can come to conclusion what could be the problem. If you suspect in the Client Hello, only a certain set of ciphers are being sent and your server is limited to a particular set of ciphers, you can check that too using ssldump.
Take a normal pcap and run ssldump to it.
Regarding the fix, once you find if its TLS 1.1 or SHA, try one after one and see the results,
#To block the weak ciphers tmsh modify ltm profile client-ssl <client-ssl-profile> ciphers "!3DES:!DH:!ADH:!EDH:!MD5:!EXPORT:!DES:!RC4:!SSLv3:!TLSv1:!TLSv1_1:!SHA1" #To block the old protocols tmsh modify ltm profile client-ssl <client-ssl-profile> options { dont-insert-empty-fragments no-sslv2 no-sslv3 no-tlsv1 no-tlsv1.1 }
Keep us posted.
- pstavrCirrus
Hi!
All tests failed today. But I have some extra clues.
- The app running on the background is a .NET application
- The registry tweaks for the .NET app basically failed. Same issue.
- We also turned off the app its self, and tried to use IIS default pages, just to see if the communication between F5 and the backend server would be OK. And it is, it works perfectly fine. So its the ,NET app that breaks it.
The weird part is that without modifying anything on F5, it works with simple IIS on the backend, but it does not with the .NET app enabled. The only difference I can see in WireShark is that the successful Client Hello done from the F5 wowards the backend server, is done using TLS 1.2. Where one that fails uses TLSv1 instead for the Client Hello. Both mark 1.0 as the minimum and 1.2 as the maximum accewpted TLS though.
Have a look at the successful attempts against IIS, and compare them to the unsuccesful against the .NET app. Only difference is that the version used for the Client Hello part. Once again, thank you all for your ideas and feedback.
Lets not stick to the record layer that it could be the issue, we should check on the other angles too (ciphers, sha, protocols).... Can you share your pcaps of failed and working one. Expand the client hello packet. Also what is there in the handshake protocol version is what would be shown in the pcap protocol info.
Also share the test results of below,
#To test tls1.1 openssl s_client -connect IP:PORT -tls1_1 #To test tls1.2 openssl s_client -connect IP:PORT -tls1_2 #To test SHA1 openssl s_client -cipher 'SHA' -connect IP:PORT -tls1_2 #To test SHA2 openssl s_client -cipher 'SHA256' -connect IP:PORT -tls1_2
- pstavrCirrus
Hi all.
I found the root cause. The problem was related to the .NET app using SNI. By default the F5 doesn't do that.
https://devcentral.f5.com/s/articles/ssl-profiles-part-7-server-name-indication
So basically I just followed the fix in the above article, I defined a server name and the backend service started sending Server Hello etc. Everything works fine now!
Thank you all for your responses, as quite a few of them were helpful on identifying that the issue is with the app, and I could also spot a few things that were not proper on the negotiation part.
Interesting, Glad your issue is resolved. One thing I learnt over the years is to look around for all possibilities. Never stick to a point and think it could be the problem.ā Mark your own answer as solution provided and close the thread.
- NUT2889Cirrostratus
Hi,
I have few experience and share with you.
Some of administrator has enabled SNI on web server e.g. IIS, Nginx, Apache. But him didn't tell me to know.
When we configure F5 to monitor via HTTP or HTTPs. The result is pool offline. Sometimes we need to re-check administrator for this setting to reduce ping-pong situation.
Actually, it's quite common, in the monitor the Host header section matters a lot.
ā
Troubleshooting health check is one thing, and troubleshooting application traffic flow is another thing.ā
Recent Discussions
Related Content
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com