Decrypting TLS traffic on BIG-IP
1 Introduction As soon as I joined F5 Support, over 5 years ago, one of the first things I had to learn quickly was to decrypt TLS traffic becausemost of our customers useL7 applications protectedby TLS layer. In this article, I will show 4 ways to decrypt traffic on BIG-IP, including thenew one just release in v15.xthat is ideal for TLS1.3 where TLS handshake is also encrypted. If that's what you want to know just skip totcpdump --f5 ssl optionsection as this new approach is just a parameter added to tcpdump. As this article is very hands-on, I will show my lab topology for the tests I performed and then every possible way I used to decrypt customer's traffic working for Engineering Services at F5. 2 Lab Topology This is the lab topology I used for the lab test where all tests were performed: Also, for every capture I issued the followingcurlcommand: Update: the virtual server's IP address is actually 10.199.3.145/32 2 The 4 ways to decrypt BIG-IP's traffic RSA private key decryption There are 3 constraints here: Full TLS handshake has to be captured CheckAppendix 2to learn how to to disable BIG-IP's cache RSA key exchange has to be used, i.e. no (EC)DHE CheckAppendix1to understand how to check what's key exchange method used in your TLS connection CheckAppendix 2to understandhow to prioritise RSA as key exchange method Private key has to be copied to Wireshark machine (ssldump command solves this problem) Roughly, to accomplish that we can setCache Sizeto 0 on SSL profile and remove (EC)DHE fromCipher Suites(seeAppendix 1for details) I first took a packet capture using :pmodifier to capture only the client and server flows specific to my Client's IP address (10.199.3.1): Note: The0.0interface will capture any forwarding plane traffic (tmm) andnnnis the highest noise to capture as much flow information as possible to be displayed on the F5 dissector header.For more details about tcpdump syntax, please have a look atK13637: Capturing internal TMM information with tcpdumpandK411: Overview of packet tracing with the tcpdump utility. Also,we need to make sure we capture the full TLS handshake. It's perfectly fine to captureresumed TLS sessionsas long as full TLS handshake has been previously captured. Initially, our capture is unencrypted as seen below: On Mac, I clicked onWireshark→Preferences: ThenProtocols→TLS→RSA keys listwhere we see a window where we can reference BIG-IP's (or server if we want to decrypt server SSL side) private key: Once we get there, we need to add any IP address of the flow we want Wireshark to decrypt, the corresponding port and the private key file (default.crtfor Client SSL profile in this particular lab test): Note:For Client SSL profile, this would be the private key onCertificate Chainfield corresponding to the end entity Certificate being served to client machines through the Virtual Server. For Server SSL profile, the private key is located on the back-end server and the key would be the one corresponding to the end entity Certificate sent in the TLSCertificatemessage sent from back-end server to BIG-IP during TLS handshake. Once we clickOK, we can see the HTTP decrypted traffic (in green): In production environment, we would normally avoid copying private keys to different machines soanother option is usessldumpcommand directly on the server we're trying to capture. Again, if we're capturing Client SSL traffic,ssldumpis already installed on BIG-IP. We would follow the same steps as before but instead of copying private key to Wireshark machine, we would simply issue this command on the BIG-IP (or back-end server if it's Server SSL traffic): Syntax:ssldump-r<capture.pcap>-k<private key.key>-M<type a name for your ssldump file here.pms>. For more details, please have a look atK10209: Overview of packet tracing with the ssldump utility. Inssldump-generated.pms, we should find enough information for Wireshark to decrypt the capture: Syntax:ssldump-r<capture.pcap>-k<private key.key>-M<type a name for your ssldump file here.pms>. For more details, please have a look atK10209: Overview of packet tracing with the ssldump utility. Inssldump-generated.pms, we should find enough information for Wireshark to decrypt the capture: After I clickedOK, we indeed see the decrypted http traffic back again: We didn't have to copy BIG-IP's private key to Wireshark machine here. iRules The only constraint here is that we should apply the iRule to the virtual server in question. Sometimes that's not desirable, especially when we're troubleshooting an issue where we want the configuration to be unchanged. Note: there is abugthat affects versions 11.6.x and 12.x that was fixed on 13.x. It records the wrong TLS Session ID to LTM logs. The workaround would be to manually copy the Session ID from tcpdump capture or to use RSA decryption as in previous example. You can also combine bothSSL::clientrandomandSSL::sessionidwhich isthe ideal: Reference: K12783074: Decrypting SSL traffic using the SSL::sessionsecret iRules command (12.x and later) Again, I took a capture usingtcpdumpcommand: After applying above iRule to our HTTPS virtual server and taking tcpdump capture, I see this on /var/log/ltm: To copy this to a *.pms file wecanuseon Wireshark we can use sed command (reference:K12783074): Note:If you don't want to overwrite completely the PMS file make sure youuse >> instead of >. The endresult would be something like this: As both resumed and full TLS sessions have client random value, I only had to copy CLIENT_RANDOM + Master secret to our PMS file because all Wireshark needs is a session reference to apply master secret. To decrypt file on Wireshark just go toWireshark→Preferences→Protocols→TLS→Pre-Master Key logfile namelike we did inssldumpsection and add file we just created: As seen on LTM logs,CLIENTSSL_HANDSHAKEevent captured master secret from our client-side connection andSERVERSSL_HANDSHAKEfrom server side. In this case, we should have both client and server sides decrypted, even though we never had access to back-end server: Notice added anhttpfilter to show you both client and and server traffic were decrypted this time. tcpdump --f5 ssl option This was introduced in 15.x and we don't need to change virtual server configuration by adding iRules. The only thing we need to do is to enabletcpdump.sslproviderdb variable which is disabled by default: After that, when we take tcpdump capture, we just need to add --f5 ssl to the command like this: Notice that we've got a warning message because Master Secret will be copied to tcpdump capture itself, so we need to be careful with who we share such capture with. I had to update my Wireshark to v3.2.+ and clicked onAnalyze→Enabled Protocols: And enable F5 TLS dissector: Once we open the capture, we can find all the information you need to create our PMS file embedded in the capture: Very cool, isn't it? We can then copy the Master Secretand Client Random values by right clicking like this: And then paste it to a blank PMS file. I first pasted the Client Random value followed by Master Secret value like this: Note: I manually typedCLIENT_RANDOMand then pasted both values for both client and server sides directly from tcpdump capture. The last step was to go toWireshark→Preferences→Protocols→TLSand add it toPre-master-Secret log filenameand clickOK: Fair enough! Capture decrypted on both client and server sides: I usedhttpfilter to display only decrypted HTTP packets just like in iRule section. Appendix 1 How do we know which Key Exchange method is being used? RSA private key can only decrypt traffic on Wireshark if RSA is the key exchange method negotiated during TLS handshake. Client side will tell the Server side which ciphers it support and server side will reply with the chosen cipher onServer Hellomessage. With that in mind, on Wireshark, we'd click onServer Helloheader underCipher Suite: Key Exchange and Authentication both come before theWITHkeyword. In above example, because there's only RSA we can say that RSA is used for both Key Exchange and Authentication. In the following example, ECDHE is used for key exchange and RSAfor authentication: Appendix 2 Disabling Session Resumption and Prioritising RSA key exchange We can set Cache Size to 0 to disableTLS session resumptionand change the Cipher Suites to anything that makes BIG-IP pick RSA for Key Exchange:Restricting number of concurrent TLS handshakes using Max Active Handshakes on BIG-IP
Introduction This is the maximum number of handshakes that BIG-IP can process concurrentlyper TMM. Max Active Handshakes can be used if there's an application specific reason to limit the number of concurrent TLS handshakes. It's typically used for performance or security reasons, i.e. when we know our application would only handle a specific number of concurrent TLS handshakes in a given time. BIG-IP allows us to restrict the number of concurrent TLS handshakes per TMM using Max Active Handshakes option on both Client SSL and Server SSL profiles: Do not confuse this setting with number of concurrent TLS connections. It is possible to have a much higher number of concurrent TLS connections than the number specified in Max Active Handshakes. We're literally talking about ongoing concurrent TLS handshakes where TLS Finished message has not been exchanged yet. Client SSL I've set max-active-handshakes to 1, which means thatBIG-IP can only process up to 1 TLS handshake at a time. From my client machine (10.199.3.135), I used the following command to test this feature and trigger 3 concurrent TLS handshakes: Note: (10.199.3.101 is the Virtual Server) Notice that there are 3 connections here and we can distinguish them by their source TCP port. First one (37062) goes through and BIG-IP (10.199.3.101) responds correctly to Client Hello (frame #11) with Server Hello (frame #14): However, the 2 subsequent Client Hello messages sent via connection 2 (37063) and connection 3 (37066) are terminated with a TSL Fatal Alert as the maximum number of allowed handshakes configured had been reached. Also notice that the Info column on Wireshark shows that all 3 connections were received by the same tmm (tmm0). Note: TMM is BIG-IP's forwarding plane daemon and may have several instances depending on how many CPU cores exist in a given box. In a subsequent test, I sent out 2 requests via 2 different interfaces concurrently to make sure another tmm was picked: Handshake completes successfully now even though max-active-handshakes is still set to 1, because each tmm (tmm0 and tmm1) processed one a handshakewhich confirms max-active-handshakes counter is per tmm: Server SSL I have also set max-active-handshakes to 1 here and issued the following command from my client machine: Note: I have changed Virtual Server to 10.199.3.145 from 10.199.3.101. Here I am filtering server-side flow only to show that only the first flow where 1st handshake was already ongoing was allowed to proceed and the 2nd handshake was not allowed: If we also include the Client-side, we see that the reset (on client-side) was because of server-side reset and reset cause isPolicy action: I have also confirmed that when max-active-handshakesset to 1, if a 2nd concurrent server-side handshake is triggered via a different TMM, then it goes through but I did not add the test here for brevity as it's the same test I performed on client-side. However, this also confirms that just like client-side, server-side setting works per TMM.Boosting TLS Performance with Dynamic Record Sizing on BIG-IP
Introduction This is what we're going to talk about today: Under normal circumstances (Allow Dynamic Record Sizing unticked on Client SSL profile), as the size of TCP Congestion Window grows to a certain size, BIG-IP typically sends TLS responses to client using a mostly fixed TLS record size. However, for long-lived connections, e.g. download of large files, TLS performance can massively increase by using a more dynamic approach with larger TLS records and BIG-IP supports that. When Allow Dynamic Record Sizing is enabled, BIG-IP dynamically adjustsoptimalTLS record layer size when forwarding traffic to client based on the state of TCP connection, i.e. TCP congestion window. How typical TLS connection behaves without Dynamic Record Sizing I captured a download of 70 MB from BIG-IP's HTTPS virtual server (10.199.3.101) to client (10.199.3.135) and here's the stats output of client-ssl profile withallow-dynamic-record-sizingdisabled: Notice we had a couple of small TLS records and then pretty much everything else was between 2-3K range. Looking at my packet capture and filtering TLS records per length/size, this is our breakdown we get: When TCP congestion window stabilises and BIG-IP starts sending data (70 MB dummy file) to client, TLS record size settles on 2920 bytes from frame #169 onwards until the end of the connection: Note: I created a new Wireshark column so we don't need to dig deeper into headers to visualise TLS record size. Before we compare above behaviour with the one we get when Dynamic Record Sizing is enabled, let's first quickly understand how Dynamic Record Sizing works. How Dynamic Record Sizing works The algorithm behaves in the following manner: When tcp congestion window is low (e.g. beginning of connection), send one TLS record per TCP packet When tcp congestion window is large AND we're transferring a large stream, then TLS record can be increased up to 16 KB to reduce framing and CPU overhead on client and server How TLS connection behaves with Dynamic Record Sizing enabled If we compare the behaviour to the one when allow-dynamic-record-sizing is enabled, we still see most TLS records on 2K - 3K range but we also see much larger records as seen below: And here's the breakdown from the capture on Wireshark: At first glance, this might not seem a massive improvement but this is just for demonstration purposes as I wanted to limit the size of my packet capture. However, the larger the file and the longer the connection time, the 5k+ TLS record size would adapt and increase accordingly. Appendix - About Lab Test Counters BIG-IP's counter is very accurate but I'd like to clarify that I used number of filtered displayed packets as the counter used for my Wireshark tables above. This is why we see some difference when compared to TMSH output as TLS records that fit into a single TCP segment will still be bundled (encapsulated) into a single TCP segment. For example, frame number #6 below has 3 TLS records of size 81, 1047 and 4 bytes respectively: Notice that I have also created a column on Wireshark to make it easier for us to visualise the TLS Record Size without having to dig deeper into headers and used a Wireshark filter (ssl.record.length >= 0 and ssl.record.length <= 1000) to display only TCP packets that contain TLS records of size > 0 and <= 1000 bytes.
