SSL protocol mismatch

[irbk]

Ok, I ended up way down a rabbit hole earlier this week.  That whole line of thought seemed to be a red herring.  BigIP LTM trying to load balance to MS Navision servers which don't use standard 80 or 443 ports.  Instead, the client communicates on port 7246 using TLSv1.2.  If I have my Virtual Server Type set to "Performance (Layer 4)" I can get a connection to the Navision servers without issue.  However I want to get SSL Bridging set up because I think we can get better performance with SSL Bridging than just the SSL passthrough (which I believe is basically what the "Performance (Layer 4)" is).  When I try to set the type to "standard" (without puting in a client or server ssl profile) the Navision client gives me a "could not create a connection to the server".  I've imported our wildcard cert and if I set the Wildcard cert for the SSL Profile (Client) and set the SSL Profile (Server) to "serverssl" I then get a "can't connect because of a protocol mismatch".  

Checking
tmm --clientciphers DEFAULT | grep "TLS1.2"
returns a bunch of TLS1.2 protocols and the Wildcard profile is set to "Ciphers Default".  Checking the LTM log, I just get kind of a generic error
Oct 4 15:45:20 BigIP01.domain.com warning tmm1[3124]: 01260009:4: <client IP>:43130 -> <BigIP VS IP>:7246: Connection error: ssl_passthru:5935: alert(40) not SSL
Now, according to wireshark, I'm seeing both TLS and non TLS traffic to port 7246 so I'm not sure if the above error is a "real" error or if the issue is because both kinds of traffic are going to the same port.  Logging on my SSL certificate is set to "debug" for all events.  I'm not sure where to go next.

ltm profile client-ssl Wildcard23-24 {
ciphers DEFAULT
}

ltm profile server-ssl serverssl {
ciphers DEFAULT
}

pool Nav_Pool_7246
profiles {
LC-http { }
LC-oneconnect { }
LC-tcp-lan { }
Wildcard23-24 {
context clientside
}
serverssl {
context serverside
}
}
serverssl-use-sni disabled
source 0.0.0.0/0
source-address-translation {
pool Nav
type snat
}
translate-address enabled
translate-port enabled
vs-index 4
}

[Amine_Kadimi]

To confirm the issue is with the TLS cipher negociation between the client and the VS, you should get a ssldump. Overview of packet tracing with the ssldump utility (f5.com)

The client hello part of the dump will show you what cipehrs are supported by the client, you can then cross check them with the 'DEFAULT' ciphers

[irbk]

This is interesting, with the SSL Profile (Client) and SSL Profile (Server) setup, doing a TCPdump capturing only the VS IP, I see 9 packets, none of which are a SSL/TLS negotation.  So it's like they aren't even trying the TLS communication or the communication is dying when they try to start the negotation. 

If I just have the SSL passthrough (IE "Performance (Layer4)") the connection succeeds and I cearly see the TLS negotation taking place.

[Amine_Kadimi]

It's not easy to tell from a tcpdump output that there is any ssl negotiation happening unless you export it to wireshark. I would definitely see what ssldump output looks like

[irbk]

Oh, yeah, I exported the TCP dump direct to wireshark and it's absoultely clear that there is no TLS communication taking place.  According to what I'm understanding @Paulius is saying below, in an SSL Bridging mode, the F5 can't handle SSL and non-SSL traffic on the same port, which is exactly how this application communicates.

[Paulius]

@irbk Some of this doesn't make sense because having a virtual server (VS) configured to listen on 443, no SSL profiles on the VS, and the pool members with port 7246 should result in a passthrough of SSL communication direclty to the pool members. Can you provide the configuration of the VS that is the standard VS without any SSL profiles?

[irbk]

Sorry @Paulius I guess I didn't post the VS configuration.  Thought I did.  The VS isn't configured to listen on 443.  It's configured to listen on 7246.  The pool members are also configured for 7246.  A wireshark of communication between the client and server with no F5 in place is what told me there is TLSv1.2 going on between the two devices on that port.

In this configuration, the client will connect without issue, because it's just doing the SSL forwarding:

ltm virtual Nav_7246 {
creation-time 2023-09-26:16:00:07
destination <VS IP>:7246
ip-protocol tcp
last-modified-time 2023-10-04:16:58:23
mask 255.255.255.255
persist {
source_addr {
default yes
}
}
pool Nav_Pool_7246
profiles {
LC-http { }
fastL4 { }
}
serverssl-use-sni disabled
source 0.0.0.0/0
source-address-translation {
pool Nav
type snat
}
translate-address enabled
translate-port enabled
vs-index 4
}

ltm pool Nav_Pool_7246 {
load-balancing-mode predictive-member
members {
Nav01:7246 {
address <Nav01 IP>
session monitor-enabled
state up
}
Nav02:7246 {
address <Nav02 IP>
session monitor-enabled
state down
}
}
monitor tcp
}

When I try with this VS configuration (same pool as above) is when I get the protocol errors

ltm virtual Nav_7246 {
creation-time 2023-09-26:16:00:07
destination <VS IP>:7246
ip-protocol tcp
last-modified-time 2023-10-05:08:35:17
mask 255.255.255.255
persist {
source_addr {
default yes
}
}
pool Nav_Pool_7246
profiles {
LC-http { }
LC-oneconnect { }
LC-tcp-lan { }
Wildcard23-24 {
context clientside
}
serverssl {
context serverside
}
}
serverssl-use-sni disabled
source 0.0.0.0/0
source-address-translation {
pool Nav
type snat
}
translate-address enabled
translate-port enabled
vs-index 4
}

[Paulius]

@irbk I'm surprised that the L4 VS configuration works since you have an HTTP profile configured on a VS that doesn't perform SSL termination or SSL bridging. For the one where you are performing SSL bridging you might try running the following tcpdump to see if can find anything odd in wireshark. You can track the connection through SNAT by filtering on the client IP source port because the F5 tends to utilize the same ephemeral port when it forms its connection from itself to the pool member that it's balancing to. This will save file "ssl_tshoot.pcap" in directory /shared/tmp/ once you stop the tcpdump.

tcpdump -nni 0.0:nnp host <vs_IP> -w /shared/tmp/ssl_tshoot.pcap

[irbk]

Actually, I just did that and replied to Amine above.  With the SSL Profile (Client) and SSL Profile (Server) setup, doing a TCPdump capturing only the VS IP, I see 9 packets, none of which are a SSL/TLS negotation.  So it's like they aren't even trying the TLS communication or the communication is dying when they try to start the negotation. 

As you can see above, no TLS communication is attempted and this is all communication between the client and VS.  The VS doesn't seem to initiate any contact with the pool member.

If I just have the SSL passthrough (IE "Performance (Layer4)") the connection succeeds and I cearly see the TLS negotation taking place.

[Paulius]

@irbk It seems as though you have both SSL and non-SSL communication occurring over the same IP and port which definitely will cause issues when perform SSL termination on the F5. The reason this will cause issues is because the F5 expects SSL communication but receives traffic that isn't SSL.

[irbk]

Yes, this is 100% correct.  Traffic starts out as just regular TCP traffic and then changes to TLS anc continues as both TLS and TCP traffic.  Here is wireshark output from a good connection (SSL passthrough).

You can see that the communication that's going on is both TCP and TLS traffic on the same port.  So are you saying the F5 is incapable of SSL bridging in this situation?

[Paulius · Accepted solution]

@irbk That is correct. Unless you have some way of the F5 being able to look for a value in the client request that would define if it was intended to be SSL or not you would have to split SSL and non-SSL into two different VS listening on different ports on the F5 side that is client facing.

[irbk]

@Paulius I don't believe there's anyway I can tell the F5 "this is TLS traffic" vs "this is TCP traffic".  I'm positive there's no way for me to tell the server to splt the traffic to two different ports.  So it kind of sounds like my only option is the SSL passthrough.  Bummer.  Thanks for the assist!