SSL Handshake failed for TCP

Hey K-Dubb

First thing you need to do is to go to Client SSL profile and disable Generic Alert.

Generic Alert hides (for security reasons) the real reason why your SSL handshake is failing.

This is good from security perspective, but for troubleshooting perspective you can safely disable it temporarily.

I would also advise to temporarily set 'Cache Size' to 0 as you will likely need to see the capture decrypted and decrypting software can only decrypt capture if it gets hold of master key and master key is only available when full TLS handshake takes place.

You can then use the following tcpdump filter to rotate through captures for every 1 GB, for example using -C flag:

$ tcpdump -i 0.0:nnn -s0 <virtual server's IP address> -C 1000 -w /shared/tmp/tls-handshake-errors.pcap -v

This would rotate the capture every 1 GB until you you have enough time to spot the TLS handshake error in the logs.

For more details about tcpdump syntax or if you want to use a different filter, you can follow K65251607 Using tcpdump to create rotating packet capture files.

Notice that you might need to have the capture decrypted so you can see the reason why TLS handshake failed.

If so, you have 2 options depending on the cipher you're using:

1 SSLdump: K10209: Overview of packet tracing with the ssldump utility

-> This would only decrypt capture when cipher chosen has RSA key exchange method

-> If you want to remove non-RSA ciphers temporarily for testing, that is fine too and you'd need to add to Cipher List the following: DEFAULT:!DHE:!ECDHE

2 iRule (Applied to Virtual Server):

when CLIENTSSL_HANDSHAKE {
 if {[IP::addr [IP::client_addr] equals TYPE_CLIENT_IP_HERE] } {
   log local0. "TCP source port: [TCP::remote_port]"
   log local0. "RSA Session-ID:[SSL::sessionid] Master-Key:[SSL::sessionsecret]" }
}

You can copy the RSA Session-ID bit to a file and load it into Wireshark.

If you want to capture traffic from a specific client the iRule works just fine regardless of the cipher used. If you need an open packet capture I would temporarily disable DHE/ECDHE and use SSLDUMP.

Please, let me know if you have further questions.

Thank you,

Rodrigo