cancel
Showing results for 
Search instead for 
Did you mean: 
George_Watkins_
Historic F5 Account

Introduction

Transport Layer Security (TLS) is used to secure network communications between two hosts. TLS largely replaced SSL (Secure Sockets Layer) starting in 1999, but many browsers still provide backwards compatibility for SSL version 3. TLS is the basis for securing all HTTPS communications on the Internet. BIG-IP provides the benefit of being able to offload the encryption and decryption of TLS traffic onto a purpose specific ASIC. This provides performance benefits for the application servers, but also provides an extra layer for troubleshooting when problems arise.

It can be a daunting task to tackle a TLS issue with tcpdump alone. Luckily, there is a utility called ssldump. Ssldump looks for TLS packets and decodes the transactions, then outputs them to the console or to a file. It will display all the components of the handshake and if a private key is provided it will also display the encrypted application data. The ability to fully examine communications from the application layer down to the network layer in one place makes troubleshooting much easier.

Note: The user interface of the BIG-IP refers to everything as SSL with little mention of TLS. The actual protocol being negotiated in these examples is TLS version 1.0, which appears as “Version 3.1” in the handshakes. For more information on the major and minor versions of TLS, see the TLS record protocol section of the Wikipedia article.

Overview of ssldump

I will spare you the man page, but here are a few of the options we will be using to examine traffic in our examples:

ssldump -A -d -k <key file> -n -i <capture VLAN> <traffic expression>

-A      Print all fields
-d      Show application data when private key is provided via -k 
-k      Private key file, found in /config/ssl/ssl.key/; the key file can be located under client SSL profile
-n      Do not try to resolve PTR records for IP addresses
-i      The capture VLAN name is the ingres VLAN for the TLS traffic

The traffic expression is nearly identical to the tcpdump expression syntax. In these examples we will be looking for HTTPS traffic between two hosts (the client and the LTM virtual server). In this case, the expression will be "host <client IP> and host <virtual server IP> and port 443”. More information on expression syntax can be found in the ssldump and tcpdump manual pages.

*the manual page can be found by typing 'man ssldump' or online here <http://www.rtfm.com/ssldump/Ssldump.html>

A healthy TLS session

When we look at a healthy TLS session we can see what things should look like in an ideal situation. First the client establishes a TCP connection to the virtual server. Next, the client initiates the handshake with a ClientHello. Within the ClientHello are a number of parameters: version, available cipher suites, a random number, and compression methods if available. The server then responds with a ServerHello in which it selects the strongest cipher suite, the version, and possibly a compression method. After these parameters have been negotiated, the server will send its certificate completing the the ServerHello. Finally, the client will respond with PreMasterSecret in the ClientKeyExchange and each will send a 1 byte ChangeCipherSpec agreeing on their symmetric key algorithm to finalize the handshake. The client and server can now exchange secure data via their TLS session until the connection is closed.

If all goes well, this is what a “clean” TLS session should look like:

New TCP connection #1: 10.0.0.10(57677) <-> 10.0.0.20(443)
1 1  0.0011 (0.0011)  C>S  Handshake
      ClientHello
        Version 3.1
        cipher suites
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        [more cipher suites]
        TLS_RSA_EXPORT_WITH_RC4_40_MD5
        Unknown value 0xff
        compression methods
                unknown value
                  NULL
1 2  0.0012 (0.0001)  S>C  Handshake
      ServerHello
        Version 3.1
        session_id[0]=

        cipherSuite         TLS_RSA_WITH_AES_256_CBC_SHA
        compressionMethod                   NULL
1 3  0.0012 (0.0000)  S>C  Handshake
      Certificate
1 4  0.0012 (0.0000)  S>C  Handshake
      ServerHelloDone
1 5  0.0022 (0.0010)  C>S  Handshake
      ClientKeyExchange
1 6  0.0022 (0.0000)  C>S  ChangeCipherSpec
1 7  0.0022 (0.0000)  C>S  Handshake
      Finished
1 8  0.0039 (0.0016)  S>C  ChangeCipherSpec
1 9  0.0039 (0.0000)  S>C  Handshake
      Finished
1 10 0.0050 (0.0010)  C>S  application_data
1    0.0093 (0.0000)  S>C  TCP FIN
1    0.0093 (0.0000)  C>S  TCP FIN

Scenario 1: Virtual server missing a client SSL profile

The client SSL profile defines what certificate and private key to use, a key passphrase if needed, allowed ciphers, and a number of other options related to TLS communications. Without a client SSL profile, a virtual server has no knowledge of any of the parameters necessary to create a TLS session. After you've configured a few hundred HTTPS virtuals this configuration step becomes automatic, but most of us mortals have missed step at one point or another and left ourselves scratching our heads.

We'll set up a test virtual that has all the necessary configuration options for an HTTPS profile, except for the omission of the client SSL profile. The client will open a connection to the virtual on port 443, a TCP connection will be established, and the client will send a 'ClientHello'. Normally the server would then respond with ServerHello, but in this case there is no response and after some period of time (5 minutes is the default timeout for the browser) the connection is closed. This is what the ssldump would look like for a missing client SSL profile:

New TCP connection #1: 10.0.0.10(46226) <-> 10.0.0.20(443)
1 1  0.0011 (0.0011)  C>SV3.1(84)  Handshake
      ClientHello
        Version 3.1
        random[32]=
          4c b6 3b 84 24 d7 93 7f 4b 09 fa f1 40 4f 04 6e
          af f7 92 e1 3b a7 3a c2 70 1d 34 dc 9d e5 1b c8
        cipher suites
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        [a number of other cipher suites]
        TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5
        TLS_RSA_EXPORT_WITH_RC4_40_MD5
        Unknown value 0xff
        compression methods
                unknown value
                  NULL
1    299.9883 (299.9871)  C>S  TCP FIN
1    299.9883 (0.0000)  S>C  TCP FIN 

Scenario 2: Client and server do not share a common cipher suite

This is a common scenario when really old browsers try to connect to servers with modern cipher suites. We have purposely configured our SSL profile to only accept one cipher suite (TLS_RSA_WITH_AES_256_CBC_SHA in this case). When we try connect to the virtual using a 128-bit key, the connection is immediately closed with no ServerHello from the virtual server. The differentiator here, while small, is the quick closure of the connection and the ‘TCP FIN’ that arises from the server. This is unlike the behavior of the missing SSL profile, because the server initiates the connection teardown and there is no connection timeout. The differences, while subtle, hint at the details of the problem:

New TCP connection #1: 10.0.0.10(49342) <-> 10.0.0.20(443)
1 1  0.0010 (0.0010)  C>SV3.1(48)  Handshake
      ClientHello
        Version 3.1
        random[32]=
          4c b7 41 87 e3 74 88 ac 89 e7 39 2d 8c 27 0d c0
          6e 27 da ea 9f 57 7c ef 24 ed 21 df a6 26 20 83
        cipher suites
        TLS_RSA_WITH_AES_128_CBC_SHA
        Unknown value 0xff
        compression methods
                unknown value
                  NULL
1    0.0011 (0.0000)  S>C  TCP FIN
1    0.0022 (0.0011)  C>S  TCP FIN

Conclusion

Troubleshooting TLS can be daunting at first, but an understanding of the TLS handshake can make troubleshooting much more approachable. We cannot exhibit every potential problem in this tech tip. However, we hope that walking through some of the more common examples will give you the tools necessary to troubleshoot other issues as they arise. Happy troubleshooting!

 
Comments
Yaoxie_117356
Nimbostratus
Nimbostratus
Not quite sure if this is related to version, but I did a quick test for scenario 1, I can see a different result. [root@V11full:Active:Standalone] config ssldump -Aed -ni 0.0 host 8.8.100.80 New TCP connection 1: 8.8.100.189(2078) <-> 8.8.100.80(443) 1 1 1433322027.9875 (0.0095) C>SV3.1(206) Handshake ClientHello Version 3.3 random[32]= dd c0 f4 f7 b7 37 88 93 6b ed b5 63 1e 94 ba 32 50 85 d3 24 d9 99 df 98 b6 84 d8 53 91 fa da c6 resume [32]= 03 1f 85 14 6b c7 9c a4 5d 21 31 3c ec 3c 4a fb 0f 6b cb 1d 31 a2 06 7b d1 9e 95 d1 3b 94 09 f4 cipher suites Unknown value 0xc02b Unknown value 0xc02f Unknown value 0xc00a Unknown value 0xc009 Unknown value 0xc013 Unknown value 0xc014 TLS_DHE_RSA_WITH_AES_128_CBC_SHA TLS_DHE_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA compression methods NULL Unknown SSL content type 72 1 2 1433322027.9891 (0.0015) S>CShort record 1 1433322027.9891 (0.0000) S>C TCP FIN 1 3 1433322027.9893 (0.0002) C>SV3.3(2) Alert level fatal value unexpected_message 1 1433322028.0057 (0.0163) C>S TCP RST In Scenario, I restricted clientssl cipher to LOW and ssldump showed [root@V11full:Active:Standalone] config ssldump -Aed -ni 0.0 host 8.8.100.80 New TCP connection 1: 8.8.100.189(2092) <-> 8.8.100.80(443) 1 1 1433322410.8985 (0.0201) C>SV3.1(174) Handshake ClientHello Version 3.3 random[32]= 0f cc 92 66 88 78 bc 53 d9 38 ad 2e cc 65 b9 25 9a 69 02 3b 55 20 dc 72 16 99 21 b2 9d 64 1f df cipher suites Unknown value 0xc02b Unknown value 0xc02f Unknown value 0xc00a Unknown value 0xc009 Unknown value 0xc013 Unknown value 0xc014 TLS_DHE_RSA_WITH_AES_128_CBC_SHA TLS_DHE_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA compression methods NULL 1 2 1433322410.8985 (0.0000) S>CV3.3(2) Alert level fatal value handshake_failure 1 1433322410.8986 (0.0000) S>C TCP FIN 1 1433322410.9205 (0.0219) C>S TCP FIN
taksh_337369
Nimbostratus
Nimbostratus

i do have one querry please if anyone help what is the meaning of 0.0 in the command

 

ssldump -ni 0.0 host 11.152.8.100 and after executing the command i am getting below output what does that means and that coming again and again .

 

New TCP connection 271: 11.152.96.2(38576) <-> 11.159.36.85(12001) New TCP connection 272: 11.152.96.2(60833) <-> 11.159.36.85(12002) 271 0.0078 (0.0078) C>S TCP FIN 272 0.0078 (0.0078) C>S TCP FIN 272 0.0088 (0.0009) S>C TCP FIN 271 0.0088 (0.0009) S>C TCP FIN New TCP connection 273: 11.152.96.2(40010) <-> 11.159.36.85(12001) New TCP connection 274: 11.152.96.2(34034) <-> 11.159.36.85(12002) 273 0.0242 (0.0242) C>S TCP FIN 274 0.0252 (0.0252) C>S TCP FIN 273 0.0252 (0.0010) S>C TCP FIN 274 0.0258 (0.0006) S>C TCP FIN

 

Gonzalex_330537
Cirrostratus
Cirrostratus

taksh, x.x are labels for the actual data / switch interfaces. Exceptionally though, 0.0 is the label for the loopback interface. In this case, it is used so that traffic is capture on all interfaces.

 

There is another exception to the rule though and that is, for eth0 interface which is the management interface.

 

Ajit
Altostratus
Altostratus

Hello George,

 

I am getting fatal ssl handshake failure(40) right after the server hello message. I can see in wireshark that the TLS protocol & ciphers are matching so not sure what else it could be. The serverssl profile is failing and the party on the other side has Citrix netscaler. We have F5 LTM at our end.Also, the citrix netscaler presents a wildcard cert to us. Could that be a problem for the F5?

 

New TCP connection 7: 10.104.41.138(56218) <-> 10.104.40.136(443) 7 1 1529673027.5089 (0.0001) C>SV3.1(121) Handshake ClientHello Version 3.3 random[32]= 46 f9 98 03 10 6c 14 84 4f 11 4e 81 f0 a0 92 dd 15 07 84 70 8c c4 94 c4 4d 2c ee 76 df d3 34 32 cipher suites Unknown value 0xc02f Unknown value 0xc030 Unknown value 0x9c Unknown value 0x9d Unknown value 0xc027 Unknown value 0xc028 Unknown value 0xc013 Unknown value 0xc014 TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA256 TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA Unknown value 0xc012 TLS_RSA_WITH_3DES_EDE_CBC_SHA Unknown value 0xff compression methods NULL 3 2 1529673027.5333 (0.0299) S>CV3.3(74) Handshake ServerHello Version 3.3 random[32]= 5b 2c f5 46 8d 5d 9a 7e 02 10 6e 1c 90 3f d6 02 cb 4c be 17 cb 7c 0c 1f 55 c8 77 fc bd 85 21 88 session_id[32]= 73 0d 48 68 8d da 73 e5 77 07 3a dc 47 a2 51 40 88 32 a2 3e d6 5c 3a 6b 4e dc c8 2c 28 d2 3c 27 cipherSuite TLS_RSA_WITH_AES_256_CBC_SHA compressionMethod NULL 3 3 1529673027.5333 (0.0000) C>SV3.3(2) Alert level fatal value handshake_failure 2 1529673027.5333 (0.0306) S>C TCP RST 3 1529673027.5334 (0.0000) C>S TCP RST

Please advise.

 

Regards,

 

Ajit

 

lossless_186906
Nimbostratus
Nimbostratus

Hi Ajit,

 

If you look closer the S>C is part of a different TCP connection (3), whereas initial C>S is 7. Maybe some asymmetric routing, missing SNAT?

 

Might be worth enabling ssl debugging and checking the logs: modify /sys db log.ssl.level value Debug

 

Cheers, Mariusz

 

Version history
Last update:
‎14-Oct-2010 11:55
Updated by:
Contributors