ABOUT DEVCENTRAL

DevCentral News Technical Forum Technical Articles Technical CrowdSRC Community Guidelines DevCentral EULA Get a Developer Lab License Become a DevCentral MVP

RESOURCES

Product Documentation White Papers Glossary Customer Stories Webinars Free Online Courses F5 Certification LearnF5 Training

SUPPORT

Manage Subscriptions Professional Services Professional Services Create a Service Request Software Downloads Support Portal

PARTNERS

Find a Reseller Partner Technology Alliances Become an F5 Partner Login to Partner Central

---

©2024 F5, Inc. All rights reserved.

Introduction

Transport Layer Security (TLS) is used to secure network communications between two hosts. TLS largely replaced SSL (Secure Sockets Layer) starting in 1999, but many browsers still provide backwards compatibility for SSL version 3. TLS is the basis for securing all HTTPS communications on the Internet. BIG-IP provides the benefit of being able to offload the encryption and decryption of TLS traffic onto a purpose specific ASIC. This provides performance benefits for the application servers, but also provides an extra layer for troubleshooting when problems arise.

It can be a daunting task to tackle a TLS issue with tcpdump alone. Luckily, there is a utility called ssldump. Ssldump looks for TLS packets and decodes the transactions, then outputs them to the console or to a file. It will display all the components of the handshake and if a private key is provided it will also display the encrypted application data. The ability to fully examine communications from the application layer down to the network layer in one place makes troubleshooting much easier.

Note: The user interface of the BIG-IP refers to everything as SSL with little mention of TLS. The actual protocol being negotiated in these examples is TLS version 1.0, which appears as “Version 3.1” in the handshakes. For more information on the major and minor versions of TLS, see the TLS record protocol section of the Wikipedia article.

Overview of ssldump

I will spare you the man page, but here are a few of the options we will be using to examine traffic in our examples:

\nssldump -A -d -k <key file> -n -i <capture VLAN> <traffic expression>\n\n-A      Print all fields\n-d      Show application data when private key is provided via -k \n-k      Private key file, found in /config/ssl/ssl.key/; the key file can be located under client SSL profile\n-n      Do not try to resolve PTR records for IP addresses\n-i      The capture VLAN name is the ingres VLAN for the TLS traffic

The traffic expression is nearly identical to the tcpdump expression syntax. In these examples we will be looking for HTTPS traffic between two hosts (the client and the LTM virtual server). In this case, the expression will be \"host <client IP> and host <virtual server IP> and port 443”. More information on expression syntax can be found in the ssldump and tcpdump manual pages.

*the manual page can be found by typing 'man ssldump' or online here <http://www.rtfm.com/ssldump/Ssldump.html>

A healthy TLS session

When we look at a healthy TLS session we can see what things should look like in an ideal situation. First the client establishes a TCP connection to the virtual server. Next, the client initiates the handshake with a ClientHello. Within the ClientHello are a number of parameters: version, available cipher suites, a random number, and compression methods if available. The server then responds with a ServerHello in which it selects the strongest cipher suite, the version, and possibly a compression method. After these parameters have been negotiated, the server will send its certificate completing the the ServerHello. Finally, the client will respond with PreMasterSecret in the ClientKeyExchange and each will send a 1 byte ChangeCipherSpec agreeing on their symmetric key algorithm to finalize the handshake. The client and server can now exchange secure data via their TLS session until the connection is closed.

If all goes well, this is what a “clean” TLS session should look like:

\nNew TCP connection #1: 10.0.0.10(57677) <-> 10.0.0.20(443)\n1 1  0.0011 (0.0011)  C>S  Handshake\n      ClientHello\n        Version 3.1\n        cipher suites\n        TLS_DHE_RSA_WITH_AES_256_CBC_SHA\n        [more cipher suites]\n        TLS_RSA_EXPORT_WITH_RC4_40_MD5\n        Unknown value 0xff\n        compression methods\n                unknown value\n                  NULL\n1 2  0.0012 (0.0001)  S>C  Handshake\n      ServerHello\n        Version 3.1\n        session_id[0]=\n\n        cipherSuite         TLS_RSA_WITH_AES_256_CBC_SHA\n        compressionMethod                   NULL\n1 3  0.0012 (0.0000)  S>C  Handshake\n      Certificate\n1 4  0.0012 (0.0000)  S>C  Handshake\n      ServerHelloDone\n1 5  0.0022 (0.0010)  C>S  Handshake\n      ClientKeyExchange\n1 6  0.0022 (0.0000)  C>S  ChangeCipherSpec\n1 7  0.0022 (0.0000)  C>S  Handshake\n      Finished\n1 8  0.0039 (0.0016)  S>C  ChangeCipherSpec\n1 9  0.0039 (0.0000)  S>C  Handshake\n      Finished\n1 10 0.0050 (0.0010)  C>S  application_data\n1    0.0093 (0.0000)  S>C  TCP FIN\n1    0.0093 (0.0000)  C>S  TCP FIN

Scenario 1: Virtual server missing a client SSL profile

The client SSL profile defines what certificate and private key to use, a key passphrase if needed, allowed ciphers, and a number of other options related to TLS communications. Without a client SSL profile, a virtual server has no knowledge of any of the parameters necessary to create a TLS session. After you've configured a few hundred HTTPS virtuals this configuration step becomes automatic, but most of us mortals have missed step at one point or another and left ourselves scratching our heads.

We'll set up a test virtual that has all the necessary configuration options for an HTTPS profile, except for the omission of the client SSL profile. The client will open a connection to the virtual on port 443, a TCP connection will be established, and the client will send a 'ClientHello'. Normally the server would then respond with ServerHello, but in this case there is no response and after some period of time (5 minutes is the default timeout for the browser) the connection is closed. This is what the ssldump would look like for a missing client SSL profile:

\nNew TCP connection #1: 10.0.0.10(46226) <-> 10.0.0.20(443)\n1 1  0.0011 (0.0011)  C>SV3.1(84)  Handshake\n      ClientHello\n        Version 3.1\n        random[32]=\n          4c b6 3b 84 24 d7 93 7f 4b 09 fa f1 40 4f 04 6e\n          af f7 92 e1 3b a7 3a c2 70 1d 34 dc 9d e5 1b c8\n        cipher suites\n        TLS_DHE_RSA_WITH_AES_256_CBC_SHA\n        [a number of other cipher suites]\n        TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5\n        TLS_RSA_EXPORT_WITH_RC4_40_MD5\n        Unknown value 0xff\n        compression methods\n                unknown value\n                  NULL\n1    299.9883 (299.9871)  C>S  TCP FIN\n1    299.9883 (0.0000)  S>C  TCP FIN 

Scenario 2: Client and server do not share a common cipher suite

This is a common scenario when really old browsers try to connect to servers with modern cipher suites. We have purposely configured our SSL profile to only accept one cipher suite (TLS_RSA_WITH_AES_256_CBC_SHA in this case). When we try connect to the virtual using a 128-bit key, the connection is immediately closed with no ServerHello from the virtual server. The differentiator here, while small, is the quick closure of the connection and the ‘TCP FIN’ that arises from the server. This is unlike the behavior of the missing SSL profile, because the server initiates the connection teardown and there is no connection timeout. The differences, while subtle, hint at the details of the problem:

\nNew TCP connection #1: 10.0.0.10(49342) <-> 10.0.0.20(443)\n1 1  0.0010 (0.0010)  C>SV3.1(48)  Handshake\n      ClientHello\n        Version 3.1\n        random[32]=\n          4c b7 41 87 e3 74 88 ac 89 e7 39 2d 8c 27 0d c0\n          6e 27 da ea 9f 57 7c ef 24 ed 21 df a6 26 20 83\n        cipher suites\n        TLS_RSA_WITH_AES_128_CBC_SHA\n        Unknown value 0xff\n        compression methods\n                unknown value\n                  NULL\n1    0.0011 (0.0000)  S>C  TCP FIN\n1    0.0022 (0.0011)  C>S  TCP FIN

Conclusion

Troubleshooting TLS can be daunting at first, but an understanding of the TLS handshake can make troubleshooting much more approachable. We cannot exhibit every potential problem in this tech tip. However, we hope that walking through some of the more common examples will give you the tools necessary to troubleshoot other issues as they arise. Happy troubleshooting!

Introduction

Transport Layer Security (TLS) is used to secure network communications between two hosts. TLS largely replaced SSL (Secure Sockets Layer) starting in 1999, but many browsers still provide backwards compatibility for SSL version 3. TLS is the basis for securing all HTTPS communications on the Internet. BIG-IP provides the benefit of being able to offload the encryption and decryption of TLS traffic onto a purpose specific ASIC. This provides performance benefits for the application servers, but also provides an extra layer for troubleshooting when problems arise.

It can be a daunting task to tackle a TLS issue with tcpdump alone. Luckily, there is a utility called ssldump. Ssldump looks for TLS packets and decodes the transactions, then outputs them to the console or to a file. It will display all the components of the handshake and if a private key is provided it will also display the encrypted application data. The ability to fully examine communications from the application layer down to the network layer in one place makes troubleshooting much easier.

Note: The user interface of the BIG-IP refers to everything as SSL with little mention of TLS. The actual protocol being negotiated in these examples is TLS version 1.0, which appears as “Version 3.1” in the handshakes. For more information on the major and minor versions of TLS, see the TLS record protocol section of the Wikipedia article.

Overview of ssldump

I will spare you the man page, but here are a few of the options we will be using to examine traffic in our examples:

\nssldump -A -d -k <key file> -n -i <capture VLAN> <traffic expression>\n\n-A      Print all fields\n-d      Show application data when private key is provided via -k \n-k      Private key file, found in /config/ssl/ssl.key/; the key file can be located under client SSL profile\n-n      Do not try to resolve PTR records for IP addresses\n-i      The capture VLAN name is the ingres VLAN for the TLS traffic

The traffic expression is nearly identical to the tcpdump expression syntax. In these examples we will be looking for HTTPS traffic between two hosts (the client and the LTM virtual server). In this case, the expression will be \"host <client IP> and host <virtual server IP> and port 443”. More information on expression syntax can be found in the ssldump and tcpdump manual pages.

*the manual page can be found by typing 'man ssldump' or online here <http://www.rtfm.com/ssldump/Ssldump.html>

A healthy TLS session

When we look at a healthy TLS session we can see what things should look like in an ideal situation. First the client establishes a TCP connection to the virtual server. Next, the client initiates the handshake with a ClientHello. Within the ClientHello are a number of parameters: version, available cipher suites, a random number, and compression methods if available. The server then responds with a ServerHello in which it selects the strongest cipher suite, the version, and possibly a compression method. After these parameters have been negotiated, the server will send its certificate completing the the ServerHello. Finally, the client will respond with PreMasterSecret in the ClientKeyExchange and each will send a 1 byte ChangeCipherSpec agreeing on their symmetric key algorithm to finalize the handshake. The client and server can now exchange secure data via their TLS session until the connection is closed.

If all goes well, this is what a “clean” TLS session should look like:

\nNew TCP connection #1: 10.0.0.10(57677) <-> 10.0.0.20(443)\n1 1  0.0011 (0.0011)  C>S  Handshake\n      ClientHello\n        Version 3.1\n        cipher suites\n        TLS_DHE_RSA_WITH_AES_256_CBC_SHA\n        [more cipher suites]\n        TLS_RSA_EXPORT_WITH_RC4_40_MD5\n        Unknown value 0xff\n        compression methods\n                unknown value\n                  NULL\n1 2  0.0012 (0.0001)  S>C  Handshake\n      ServerHello\n        Version 3.1\n        session_id[0]=\n\n        cipherSuite         TLS_RSA_WITH_AES_256_CBC_SHA\n        compressionMethod                   NULL\n1 3  0.0012 (0.0000)  S>C  Handshake\n      Certificate\n1 4  0.0012 (0.0000)  S>C  Handshake\n      ServerHelloDone\n1 5  0.0022 (0.0010)  C>S  Handshake\n      ClientKeyExchange\n1 6  0.0022 (0.0000)  C>S  ChangeCipherSpec\n1 7  0.0022 (0.0000)  C>S  Handshake\n      Finished\n1 8  0.0039 (0.0016)  S>C  ChangeCipherSpec\n1 9  0.0039 (0.0000)  S>C  Handshake\n      Finished\n1 10 0.0050 (0.0010)  C>S  application_data\n1    0.0093 (0.0000)  S>C  TCP FIN\n1    0.0093 (0.0000)  C>S  TCP FIN

Scenario 1: Virtual server missing a client SSL profile

The client SSL profile defines what certificate and private key to use, a key passphrase if needed, allowed ciphers, and a number of other options related to TLS communications. Without a client SSL profile, a virtual server has no knowledge of any of the parameters necessary to create a TLS session. After you've configured a few hundred HTTPS virtuals this configuration step becomes automatic, but most of us mortals have missed step at one point or another and left ourselves scratching our heads.

We'll set up a test virtual that has all the necessary configuration options for an HTTPS profile, except for the omission of the client SSL profile. The client will open a connection to the virtual on port 443, a TCP connection will be established, and the client will send a 'ClientHello'. Normally the server would then respond with ServerHello, but in this case there is no response and after some period of time (5 minutes is the default timeout for the browser) the connection is closed. This is what the ssldump would look like for a missing client SSL profile:

\nNew TCP connection #1: 10.0.0.10(46226) <-> 10.0.0.20(443)\n1 1  0.0011 (0.0011)  C>SV3.1(84)  Handshake\n      ClientHello\n        Version 3.1\n        random[32]=\n          4c b6 3b 84 24 d7 93 7f 4b 09 fa f1 40 4f 04 6e\n          af f7 92 e1 3b a7 3a c2 70 1d 34 dc 9d e5 1b c8\n        cipher suites\n        TLS_DHE_RSA_WITH_AES_256_CBC_SHA\n        [a number of other cipher suites]\n        TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5\n        TLS_RSA_EXPORT_WITH_RC4_40_MD5\n        Unknown value 0xff\n        compression methods\n                unknown value\n                  NULL\n1    299.9883 (299.9871)  C>S  TCP FIN\n1    299.9883 (0.0000)  S>C  TCP FIN 

Scenario 2: Client and server do not share a common cipher suite

This is a common scenario when really old browsers try to connect to servers with modern cipher suites. We have purposely configured our SSL profile to only accept one cipher suite (TLS_RSA_WITH_AES_256_CBC_SHA in this case). When we try connect to the virtual using a 128-bit key, the connection is immediately closed with no ServerHello from the virtual server. The differentiator here, while small, is the quick closure of the connection and the ‘TCP FIN’ that arises from the server. This is unlike the behavior of the missing SSL profile, because the server initiates the connection teardown and there is no connection timeout. The differences, while subtle, hint at the details of the problem:

\nNew TCP connection #1: 10.0.0.10(49342) <-> 10.0.0.20(443)\n1 1  0.0010 (0.0010)  C>SV3.1(48)  Handshake\n      ClientHello\n        Version 3.1\n        random[32]=\n          4c b7 41 87 e3 74 88 ac 89 e7 39 2d 8c 27 0d c0\n          6e 27 da ea 9f 57 7c ef 24 ed 21 df a6 26 20 83\n        cipher suites\n        TLS_RSA_WITH_AES_128_CBC_SHA\n        Unknown value 0xff\n        compression methods\n                unknown value\n                  NULL\n1    0.0011 (0.0000)  S>C  TCP FIN\n1    0.0022 (0.0011)  C>S  TCP FIN

Conclusion

Troubleshooting TLS can be daunting at first, but an understanding of the TLS handshake can make troubleshooting much more approachable. We cannot exhibit every potential problem in this tech tip. However, we hope that walking through some of the more common examples will give you the tools necessary to troubleshoot other issues as they arise. Happy troubleshooting!

Hi Ajit,

If you look closer the S>C is part of a different TCP connection (3), whereas initial C>S is 7. Maybe some asymmetric routing, missing SNAT?

Might be worth enabling ssl debugging and checking the logs:\nmodify /sys db log.ssl.level value Debug

Cheers,\nMariusz

Hello George,

I am getting fatal ssl handshake failure(40) right after the server hello message. I can see in wireshark that the TLS protocol & ciphers are matching so not sure what else it could be. The serverssl profile is failing and the party on the other side has Citrix netscaler. We have F5 LTM at our end.Also, the citrix netscaler presents a wildcard cert to us. Could that be a problem for the F5?

\nNew TCP connection 7: 10.104.41.138(56218) <-> 10.104.40.136(443)\n7 1  1529673027.5089 (0.0001)  C>SV3.1(121)  Handshake\n      ClientHello\n        Version 3.3 \n        random[32]=\n          46 f9 98 03 10 6c 14 84 4f 11 4e 81 f0 a0 92 dd \n          15 07 84 70 8c c4 94 c4 4d 2c ee 76 df d3 34 32 \n        cipher suites\n        Unknown value 0xc02f\n        Unknown value 0xc030\n        Unknown value 0x9c\n        Unknown value 0x9d\n        Unknown value 0xc027\n        Unknown value 0xc028\n        Unknown value 0xc013\n        Unknown value 0xc014\n        TLS_RSA_WITH_AES_128_CBC_SHA256\n        TLS_RSA_WITH_AES_256_CBC_SHA256\n        TLS_RSA_WITH_AES_128_CBC_SHA\n        TLS_RSA_WITH_AES_256_CBC_SHA\n        Unknown value 0xc012\n        TLS_RSA_WITH_3DES_EDE_CBC_SHA\n        Unknown value 0xff\n        compression methods\n                  NULL\n3 2  1529673027.5333 (0.0299)  S>CV3.3(74)  Handshake\n      ServerHello\n        Version 3.3 \n        random[32]=\n          5b 2c f5 46 8d 5d 9a 7e 02 10 6e 1c 90 3f d6 02 \n          cb 4c be 17 cb 7c 0c 1f 55 c8 77 fc bd 85 21 88 \n        session_id[32]=\n          73 0d 48 68 8d da 73 e5 77 07 3a dc 47 a2 51 40 \n          88 32 a2 3e d6 5c 3a 6b 4e dc c8 2c 28 d2 3c 27 \n        cipherSuite         TLS_RSA_WITH_AES_256_CBC_SHA\n        compressionMethod                   NULL\n3 3  1529673027.5333 (0.0000)  C>SV3.3(2)  Alert\n    level           fatal\n    value           handshake_failure\n2    1529673027.5333 (0.0306)  S>C  TCP RST\n3    1529673027.5334 (0.0000)  C>S  TCP RST\n\n

Please advise.

Regards,

Ajit

taksh, x.x are labels for the actual data / switch interfaces. Exceptionally though, 0.0 is the label for the loopback interface. In this case, it is used so that traffic is capture on all interfaces.

There is another exception to the rule though and that is, for eth0 interface which is the management interface.

i do have one querry please if anyone help what is the meaning of 0.0 in the command

ssldump -ni 0.0 host 11.152.8.100 and after executing the command i am getting below output what does that means and that coming again and again .

New TCP connection 271: 11.152.96.2(38576) <-> 11.159.36.85(12001)\nNew TCP connection 272: 11.152.96.2(60833) <-> 11.159.36.85(12002)\n271 0.0078 (0.0078) C>S TCP FIN\n272 0.0078 (0.0078) C>S TCP FIN\n272 0.0088 (0.0009) S>C TCP FIN\n271 0.0088 (0.0009) S>C TCP FIN\nNew TCP connection 273: 11.152.96.2(40010) <-> 11.159.36.85(12001)\nNew TCP connection 274: 11.152.96.2(34034) <-> 11.159.36.85(12002)\n273 0.0242 (0.0242) C>S TCP FIN\n274 0.0252 (0.0252) C>S TCP FIN\n273 0.0252 (0.0010) S>C TCP FIN\n274 0.0258 (0.0006) S>C TCP FIN