cancel
Showing results for 
Search instead for 
Did you mean: 

1 Introduction

As soon as I joined F5 Support, over 5 years ago, one of the first things I had to learn quickly was to decrypt TLS traffic because most of our customers useL7 applications protected by TLS layer.

In this article, I will show 4 ways to decrypt traffic on BIG-IP, including the new one just release in v15.x that is ideal for TLS1.3 where TLS handshake is also encrypted.

If that's what you want to know just skip to tcpdump --f5 ssl option section as this new approach is just a parameter added to tcpdump.

As this article is very hands-on, I will show my lab topology for the tests I performed and then every possible way I used to decrypt customer's traffic working for Engineering Services at F5.

2 Lab Topology

This is the lab topology I used for the lab test where all tests were performed:

0151T000003WHBJQA4.png

Also, for every capture I issued the following curl command:

0151T000003WHBKQA4.png

Update: the virtual server's IP address is actually 10.199.3.145/32

2 The 4 ways to decrypt BIG-IP's traffic

RSA private key decryption


0151T000003WHBOQA4.png

There are 3 constraints here:

  • Full TLS handshake has to be captured
  • Check Appendix 2 to learn how to to disable BIG-IP's cache
  • RSA key exchange has to be used, i.e. no (EC)DHE
  • Check Appendix 1 to understand how to check what's key exchange method used in your TLS connection
  • Check Appendix 2 to understand how to prioritise RSA as key exchange method
  • Private key has to be copied to Wireshark machine (ssldump command solves this problem)

Roughly, to accomplish that we can set Cache Size to 0 on SSL profile and remove (EC)DHE from Cipher Suites (see Appendix 1 for details)

I first took a packet capture using :p modifier to capture only the client and server flows specific to my Client's IP address (10.199.3.1😞

0151T000003WHBTQA4.png

Note: The 0.0 interface will capture any forwarding plane traffic (tmm) and nnn is the highest noise to capture as much flow information as possible to be displayed on the F5 dissector header. For more details about tcpdump syntax, please have a look at K13637: Capturing internal TMM information with tcpdump and K411: Overview of packet tracing with the tcpdump utility. Also, we need to make sure we capture the full TLS handshake. It's perfectly fine to capture resumed TLS sessions as long as full TLS handshake has been previously captured.

Initially, our capture is unencrypted as seen below:

0151T000003WHBPQA4.png

On Mac, I clicked on Wireshark → Preferences:

0151T000003WHBUQA4.png

Then Protocols → TLS → RSA keys list where we see a window where we can reference BIG-IP's (or server if we want to decrypt server SSL side) private key:

0151T000003WHBYQA4.gif

Once we get there, we need to add any IP address of the flow we want Wireshark to decrypt, the corresponding port and the private key file (default.crt for Client SSL profile in this particular lab test):

Note: For Client SSL profile, this would be the private key on Certificate Chain field corresponding to the end entity Certificate being served to client machines through the Virtual Server. For Server SSL profile, the private key is located on the back-end server and the key would be the one corresponding to the end entity Certificate sent in the TLS Certificate message sent from back-end server to BIG-IP during TLS handshake.

0151T000003WHBQQA4.gif

Once we click OK, we can see the HTTP decrypted traffic (in green😞

0151T000003WHBZQA4.gif

In production environment, we would normally avoid copying private keys to different machines so another option is use ssldump command directly on the server we're trying to capture.

Again, if we're capturing Client SSL traffic, ssldump is already installed on BIG-IP.

We would follow the same steps as before but instead of copying private key to Wireshark machine, we would simply issue this command on the BIG-IP (or back-end server if it's Server SSL traffic):

0151T000003WHBVQA4.png

Syntaxssldump -r <capture.pcap> -k <private key.key> -M <type a name for your ssldump file here.pms> . For more details, please have a look at K10209: Overview of packet tracing with the ssldump utility.

In ssldump-generated.pms, we should find enough information for Wireshark to decrypt the capture:

0151T000003WHBaQAO.png

Syntaxssldump -r <capture.pcap> -k <private key.key> -M <type a name for your ssldump file here.pms> . For more details, please have a look at K10209: Overview of packet tracing with the ssldump utility.

In ssldump-generated.pms, we should find enough information for Wireshark to decrypt the capture:

0151T000003WHBWQA4.png

After I clicked OK, we indeed see the decrypted http traffic back again:

0151T000003WHBXQA4.png

We didn't have to copy BIG-IP's private key to Wireshark machine here.

iRules

0151T000003WHBdQAO.png

The only constraint here is that we should apply the iRule to the virtual server in question.

Sometimes that's not desirable, especially when we're troubleshooting an issue where we want the configuration to be unchanged.

Note: there is a bug that affects versions 11.6.x and 12.x that was fixed on 13.x. It records the wrong TLS Session ID to LTM logs. The workaround would be to manually copy the Session ID from tcpdump capture or to use RSA decryption as in previous example.

You can also combine both SSL::clientrandom and SSL::sessionid which is the ideal:

0151T000003WHBiQAO.png

Reference: K12783074: Decrypting SSL traffic using the SSL::sessionsecret iRules command (12.x and later)

Again, I took a capture using tcpdump command:

0151T000003WHBbQAO.png

After applying above iRule to our HTTPS virtual server and taking tcpdump capture, I see this on /var/log/ltm:

0151T000003WHBeQAO.png

To copy this to a *.pms file we can use on Wireshark we can use sed command (reference: K12783074😞

0151T000003WHBnQAO.png

Note: If you don't want to overwrite completely the PMS file make sure you use >> instead of >.

The end result would be something like this:

0151T000003WHBoQAO.png

As both resumed and full TLS sessions have client random value, I only had to copy CLIENT_RANDOM + Master secret to our PMS file because all Wireshark needs is a session reference to apply master secret.

To decrypt file on Wireshark just go to Wireshark →  Preferences → Protocols → TLS → Pre-Master Key logfile name like we did in ssldump section and add file we just created:

0151T000003WHBcQAO.png

As seen on LTM logs, CLIENTSSL_HANDSHAKE event captured master secret from our client-side connection and SERVERSSL_HANDSHAKE from server side.

In this case, we should have both client and server sides decrypted, even though we never had access to back-end server:

0151T000003WHBkQAO.png

Notice added an http filter to show you both client and and server traffic were decrypted this time.

tcpdump --f5 ssl option

This was introduced in 15.x and we don't need to change virtual server configuration by adding iRules.

The only thing we need to do is to enable tcpdump.sslprovider db variable which is disabled by default:

0151T000003WHBlQAO.png

After that, when we take tcpdump capture, we just need to add --f5 ssl to the command like this:

0151T000003WHBLQA4.png

Notice that we've got a warning message because Master Secret will be copied to tcpdump capture itself, so we need to be careful with who we share such capture with.

I had to update my Wireshark to v3.2.+ and clicked on Analyze  → Enabled Protocols:

0151T000003WHBsQAO.png

And enable F5 TLS dissector:

0151T000003WHBpQAO.png

Once we open the capture, we can find all the information you need to create our PMS file embedded in the capture:

0151T000003WHBxQAO.png

Very cool, isn't it?

We can then copy the Master Secret and Client Random values by right clicking like this:

0151T000003WHC7QAO.png

And then paste it to a blank PMS file.

I first pasted the Client Random value followed by Master Secret value like this:

0151T000003WHCCQA4.png

Note: I manually typed CLIENT_RANDOM and then pasted both values for both client and server sides directly from tcpdump capture.

The last step was to go to  Wireshark →  Preferences →  Protocols → TLS and add it to Pre-master-Secret log filename and click OK:

0151T000003WHCHQA4.png

Fair enough! Capture decrypted on both client and server sides:

0151T000003WHBfQAO.png

I used http filter to display only decrypted HTTP packets just like in iRule section.

Appendix 1 How do we know which Key Exchange method is being used?

RSA private key can only decrypt traffic on Wireshark if RSA is the key exchange method negotiated during TLS handshake.

Client side will tell the Server side which ciphers it support and server side will reply with the chosen cipher on Server Hello message.

With that in mind, on Wireshark, we'd click on Server Hello header under Cipher Suite:

0151T000003WHCIQA4.png

Key Exchange and Authentication both come before the WITH keyword.

In above example, because there's only RSA we can say that RSA is used for both Key Exchange and Authentication.

In the following example, ECDHE is used for key exchange and RSA for authentication:

0151T000003WHBtQAO.png

Appendix 2 Disabling Session Resumption and Prioritising RSA key exchange

We can set Cache Size to 0 to disable TLS session resumption and change the Cipher Suites to anything that makes BIG-IP pick RSA for Key Exchange:

0151T000003WHCDQA4.png

Comments
Dojs
Cirrostratus
Cirrostratus

Great article!

Walter_Kacynski
Cirrostratus
Cirrostratus

What is the purpose of "Generate KEYLOG records from TLS f5ethtrailer" option for f5ethtrailer disector?

keylog is the file you're going to create with the value shown in the F5 TLS headers. You can copy what's after "[Keylog entry:" as is and paste it to a file, then go to Wireshark preferences -> Protocols -> TLS and add this file as pre-master key. It should decrypt your capture.

Walter_Kacynski
Cirrostratus
Cirrostratus

I manually reconstructed the file per these directions with success, but I'm still not finding the "[Keylog" text you are referring to. When there are a large number of TCP handshakes, it would be nice to have a way to extract all this data with a script.

If you scroll down you should eventually find it. You can create your own script with Python/bash using tshark to extract it and generate the file automatically.

Walter_Kacynski
Cirrostratus
Cirrostratus

I am running WS 3.2.3 on Windows 64bit and that data is not populating. I see that the "Type" field is a different version as well. This capture was done under version 15.1.0.20691T000008t35fQAA.png

Not all packets will show complete keyloggeroutput. I'm also using Wireshark v3.2.3. Even if for some reason it's not visible, you can just create a file with CLIENT_RANDOM <client random value> <master key value> and it should be enough to decrypt capture.

rob_carr
MVP
MVP

Hi Rodrigo,

 

Really great article. I've been wondering how to decrypt both client-side and server-side when using the 'p' modifier with tcpdump, and this article does a nice job of explaining exactly how to do it. I'm going to recommend this article to coworkers.

 

 

 

Chris_Zhang
F5 Employee
F5 Employee

Thanks Rodrigo! What a great article you have written! Quality stuff!

 

One clarification:

 

Note: I manually typed CLIENT_RANDOM and then pasted both values for both client and server sides directly from tcpdump capture.

 

Is 'Server Random' needed? Or just,

 

CLIENT_RANDOM $client_random $master_secret

Chris,

 

Thanks a lot! I'm glad you found it helpful.

 

You don't need Server Random, only CLIENT_RANDOM $client_random $master_secret

 

However, note that F5 makes it easier for you by pasting exactly what you need to paste to your *.pms file on "Keylog entry" field as seen below:

0691T000009iuQfQAI.png

Version history
Last update:
‎23-Apr-2020 04:23
Updated by:
Contributors