SNI Routing with BIG-IP
In the previous article, The Three HTTP Routing Patterns, Lori MacVittie covers 3 methods of routing. Today we will look at Server Name Indication (SNI) routing as an additional method of routing HTTPS or any protocol that uses TLS and SNI. Using SNI we can route traffic to a destination without having to terminate the SSL connection. This enables several benefits including: Reduced number of Public IPs Simplified configuration More intelligent routing of TLS traffic Terminating SSL Connections When you havea SSL certificate and key you can perform the cryptographic actions required to encrypt traffic using TLS. This is what I refer to as “terminating the SSL connection” throughout this article. When you want to route traffic this is a chickenand an egg problem, becausefor TLS traffic you want to be able to route the traffic by being able to inspect the contents, but this normally requires being able to “terminate the SSL connection”. The goal of this article is to layer in traffic routing for TLS traffic without having to require having/knowing the original SSL certificate and key. Server Name Indication (SNI) SNI is a TLS extension that makes it possible to "share" certificates on a single IP address. This is possible due to a client using a TLS extension that requests a specific name before the server responds with a SSL certificate. Prior to SNI, the other options would be a wildcard certificate or Subject Alternative Name (SAN) that allows you to specify multiple names with a single certificate. SNI with Virtual Servers It has been possible to use SNI on F5 BIG-IP since TMOS 11.3.0. The following KB13452 outlines how it can be configured. In this scenario (from the KB article) the BIG-IP is terminating the SSL connection. Not all clients support SNI and you will always need to specify a “fallback” profile that will be used if a SNI name is not used or matched. The next example will look at how to use SNI without terminating the SSL connection. SNI Routing Occasionally you may have the need to have a hybrid configuration of terminating SSL connections on the BIG-IP and sending connections without terminating SSL. One method is to create two separate virtual servers, one for SSL connections that the BIG-IP will handle (using clientssl profile) and one that it will not handle SSL (just TCP). This works OK for a small number of backends, but does not scale well if you have many backends (run out of Public IP addresses). Using SNI Routing we can handle everything on a single virtual server / Public IP address. There are 3 methods for performing SNI Routing with BIG-IP iRule with binary scan a. Article by Colin Walker code attribute to Joel Moses b. Code Share by Stanislas Piron iRule with SSL::extensions Local Traffic Policy Option #1 is for folks that prefer complete control of the TLS protocol. It only requires the use of a TCP profile. Options #2 and #3 only require the use of a SSL persistence profile and TCP profile. SNI Routing with Local Traffic Policy We will skip option #1 and #2 in this article and look at using a Local Traffic Policy for SNI Routing. For a review of Local Traffic Policies check out the following DevCentral articles: LTM Policy Jan 2015 Simplifying Local Traffic Policies in BIG-IP 12.1 June 2016 In previous articles about Local Traffic Policiesthe focus was on routing HTTP traffic, but today we will use it to route SSL connections using SNI. In the following example, using a Local Traffic Policy named “sni_routing”, we are setting a condition on the SSL Extension “servername” and sending the traffic to a pool without terminating the SSL connection. The pool member could be another server or another BIG-IP device. The next example will forward the traffic to another virtual server that is configured with a clientssl profile. This uses VIP targeting to send traffic to another virtual server on the same device. In both examples it is important to note that the “condition”/“action” has been changed from occurring on “request” (that maps to a HTTP L7 request) to “ssl client hello”. By performing the action prior to any L7 functions occurring, we can forward the traffic without terminating the SSL connection. The previous example policy, “sni_routing”, can be attached to a Virtual Server that only has a TCP profile and SSL persistence profile. No HTTP or clientssl profile is required! This method can also be used to solve the issue of how to consolidate multiple SSL virtual servers behind a single virtual server that have different APM and/or ASM policies. This is similar to the architecture that is used by the Container Connector for Cloud Foundry; in creating a two-tier load balancing solution on a single device. Routed Correctly? TLS 1.3 has interesting proposals on how to obscure the servername (TLS in TLS?), but for now this is a useful and practical method of handling multiple SSL certs on a single IP. In the future this may still be possible as well with TLS 1.3. For example the use of a HTTP Fronting service could be a tier 1 virtual server (this is just my personal speculation, I have not tried, at the time of publishing this was still a draft proposal). In other news it has been demonstrated that a combination of using SNI and a different host header can be used for “domain fronting”. A method to enforce consistent policy (prevent domain fronting) would be to layer in additional conditions that match requested SNI servername (TLS extension) with requested HOST header (L7 HTTP header). This would help enforce that a tenant is using a certificate that is associated with their application and not “borrowing” the name and certificate that is being used by an adjacent service. We don’t think of a TLS extension as an attribute that can be used to route application traffic, but it is useful and possible on BIG-IP.
Client SSL Authentication on BIG-IP as in-depth as it can go
Quick Intro In this article, I'm going to explain how SSL client certificate authentication works on BIG-IP and explain what actually happens during client authentication as in-depth as I can, showing the TLS headers on Wireshark. This article is about the client side of BIG-IP (Client SL profile) authenticating a client connecting to BIG-IP. The Topology For reference so we can follow Wireshark output: How to Configure Client Certificate Authentication on Client SSL profile Essentially, what we're doing here is making BIG-IP verify client's credentials before allowing the TLS handshake to proceed. However, such credentials are in the form of a client certificate. The way to do this is to configure BIG-IP by: Adding a CA file to Trusted Certificate Authorities (ca-file in tmsh) to validate client certificate Optionally adding same CA that signed client certificate to Advertised Certificate Authorities Enforcing Client Certificate validation by setting Client Certificate option on BIG-IP to require Optionally setting the Frequency of such checks if we don't want to stick to the defaults. I'll go through each option now. Adding CA file to Trusted Certificate Authorities We should add to Trusted Certificate Authorities a single certificate file (*.crt) with one CA or concatenated file with 2 or more CAs with the purpose of validating client certificate, i.e. confirming client's identity. Upon receiving client certificate, BIG-IP will go through this list of CAs and confirm client's identity. It also has another purpose which is to authenticate BIG-IP to client but it's out of the scope of this article. Optionally add CA file to Advertised Certificate Authorities Trusted Certificate Authoritiesexplicitly tells BIG-IP the CA or chain of CAs it will use to validate client certificate whereasAdvertised Certificate Authorities tells client in advance what kind of CA BIG-IP trusts so that client can make the decision about which certificate to send to BIG-IP: Why would we use Advertised Certificate Authorities? Let's imagine a situation where Client's application has more than one Client Certificate configured. How is it going to figure out which certificate to send to BIG-IP? That's where Advertised Certificate Authorities comes to rescue us! When we add our CA bundle to Advertised Certificate Authorities, we're telling BIG-IP to add it to a header field named Distinguished Names within Certificate Request message. I dedicated the Appendix section to show you more in-depth how changing Advertised Certificate Authority affects Certificate Request header. Configuring BIG-IP to enforce Client Certificate validation To enable client certificate authentication on BIG-IP we change Local Traffic››Profiles : SSL : Client›› Client Certificate to request: The default is set to ignore where client certificate authentication is disabled. If we truly want to enable client certificate validation we need to select require. The reason why is because request makes BIG-IP request a client certificate from the client but BIG-IP will not perform the validation to confirm if the certificate sent is valid in this mode. The following subsections explain each option. Ignore Client Certificate Authentication is disabled (the default). BIG-IP never sends Certificate Request to client and therefore client does not need to send its certificate to BIG-IP. In this case, TLS handshake proceeds successfully without any client authentication: pcap:ssl-sample-peer-cert-mode-ignore.pcap Wireshark filter used:frame.number == 5 or frame.number == 6 Request BIG-IP requests Client Certificate by sending Certificate Request message but does not check whether client certificate is valid which is not really client authentication, is it? This means that ca-file(Trusted Certificate Authorities in the GUI) will not be used to validate client certificate and we will consider any certificate sent to us to be valid. For example, inssl-sample-peer-cert-mode-request-with-no-client-cert-sent.pcapwe now see that BIG-IP sends aCertificate Requestmessage and client responds with aCertificatemessage this time: Because I didn't add any client certificate to my browser, it sent a blank certificate to BIG-IP. Again, BIG-IP did not perform any validation whatsoever, so TLS handshake proceeded successfully. We can then conclude that this setting only makes BIG-IP request client certificate and that's it. Require It behaves just like Request but BIG-IP also performs client certificate validation, i.e. BIG-IP will use CA we hopefully added to ca-file (Trusted Certificate Authorities in the GUI) to confirm if client certificate is valid. This means that if we don't add a CA to Trusted Certificate Authorities (ca-file in tmsh) then validation will fail. Setting the Frequency of Client Certificate Requests This setting specifies the frequency BIG-IP authenticates client by enabling/disabling TLS session resumption. It has only 2 options: once BIG-IP requests client certificate during first handshake and no longer re-authenticates client as long as TLS session is reused and valid. The way BIG-IP does it is by using Session Resumption/Reuse. During first TLS handshake from client, BIG-IP sends a Session ID to Client within Server Hello header and in subsequent TLS connections, assuming session ID is still in BIG-IP's cache and client re-sends it back to BIG-IP, then session will be resumed every time client tries to establish a TLS session (respecting cache timeout). First time client sends Client Hello with blank session ID as it's cache is empty and then it is assigned a Session ID by BIG-IP (409f...): pcap:ssl-sample-clientcert-auth-once-enabled.pcap Wireshark filter used:filter:!ip.addr == 172.16.199.254 and frame.number > 1 and frame.number < 7 Certificate Requestconfirms BIG-IP is trying to authenticate client. Notice Session ID BIG-IP sent to client is 409f7... Then when Client tries to go through another TLS handshake and sends above session ID in Client Hello (packet #70 below). BIG-IP then confirms session is being resumed by sending samesession IDin Server Hello back to client: Wireshark filter used:!ip.addr == 172.16.199.254 and frame.number > 66 and frame.number < 73 This resumed TLS handshake just means we will not go through full handshake and will no longer need to exchange keys, select ciphers or re-authenticate client as we're reusing those already negotiated in the full TLS handshake where we first received Session ID 409f... always BIG-IP requests client certificate, i.e. re-authenticates client at every handshake. On BIG-IP, this is accomplished by disabling session reuse which makes BIG-IP not to sendSession IDback to Client in the beginning and forcing a full TLS handshake every time. pcap:ssl-sample-clientcert-auth-always-enabled.pcap Wireshark filter used: !ip.addr == 172.16.199.254 and ((frame.number > 1 and frame.number < 7) or (frame.number > 74 and frame.number < 80)) Appendix: Understanding how Advertised Certificate Authority field affects Certificate Request header For this test, I've got the following: myCAbundle.crt: concatenation of root_ca.crt and ltm2.crt (signed by root_ca.crt) client_cert.crt: added to Firefox and signed by ltm2.crt I've also added myCAbundle.crt to Trusted Certificate Authorities so BIG-IP is able to verify that client_cert.crt is valid. For each test, I will use change Advertised Certificate Authorities so we can see what happens. We'll go through 3 tests here: Setting Advertised Certificate Authority to None Setting Advertised Certificate Authority to a certificate that didn't sign client cert Setting Advertised Certificate Authority to a bundle that signed client cert Setting Advertised Certificate Authority to None Note that even though no CA was advertised in Certificate Request message, BIG-IP still advertises Certificate typesandSignature Hash Algorithmsso that client knows in advance what kind of certificate (RSA, DSS or ECDSA) and hash algorithms BIG-IP supports in advance. If client certificate had not been signed using any of the certificate types and hashing algorithms listed then handshake would have failed. However, in this case validation is successful as we can see on frame #8 that client certificate is RSA type and hashed with SHA1: pcap: ssl-sample-advcert-none.pcap filter used: !ip.addr == 172.16.199.254 and frame.number > 1 and frame.number < 16 It's worth noting that Distinguished Names is NOT populated and has length of zero because we didn't attach a bundle to Advertised Certificate Authorities. In this case, it worked fine because my client browser had only one certificate attached, so it shouldn't cause a problem anyway. Setting Advertised Certificate Authority to a certificate that didn't sign client cert pcap: ssl-sample-advcert-default-firefox.pcap Wireshark filter used: None I've set Advertised Certificate Authority to default.crt as this is NOT the CA that signed client's certificate: The difference here when compared to None is that now we can see that Distinguished Names is now populated with the certificate I added (default.crt): However, even though I added the correct certificate to my Firefox browser, it sent a blank certificate instead. Why? That's because BIG-IP signalled in Distinguished Names that default.crt is the CA that signed the certificate BIG-IP is looking for and as Firefox doesn't have any certificate signed by default.crt, it just sent a blank certificate back to BIG-IP. Also, because BIG-IP is now performing proper validation, i.e. comparing whatever client certificate is sent to it with the CA list added to Trusted Certificate Authorities, it knows a blank certificate is not valid and terminates the TLS handshake with a Fatal Alert. Setting Advertised Certificate Authority to a bundle that signed client cert pcap: ssl-sample-advcert-ltm2chainedwithrootca.pcap filter used: !ip.addr == 172.16.199.254 and frame.number > 1 Now I've set Advertised Certificate Authority to the correct bundle that signed my client certificate: And indeed handshake succeeds because: BIG-IP advertises myCAbundle.crt in Certificate Request >> Distinguished Names header, as per Advertised Certificate Authority configuration By reading Distinguished Names field, Client correctly sends the correct client certificate back to BIG-IP BIG-IP correctly validates client certificate using myCAbundle configured on Trusted Certificate Authorities Hope this article provides some clarification about these mysterious TLS headers.
Decrypting TLS traffic on BIG-IP
1 Introduction As soon as I joined F5 Support, over 5 years ago, one of the first things I had to learn quickly was to decrypt TLS traffic becausemost of our customers useL7 applications protectedby TLS layer. In this article, I will show 4 ways to decrypt traffic on BIG-IP, including thenew one just release in v15.xthat is ideal for TLS1.3 where TLS handshake is also encrypted. If that's what you want to know just skip totcpdump --f5 ssl optionsection as this new approach is just a parameter added to tcpdump. As this article is very hands-on, I will show my lab topology for the tests I performed and then every possible way I used to decrypt customer's traffic working for Engineering Services at F5. 2 Lab Topology This is the lab topology I used for the lab test where all tests were performed: Also, for every capture I issued the followingcurlcommand: Update: the virtual server's IP address is actually 10.199.3.145/32 2 The 4 ways to decrypt BIG-IP's traffic RSA private key decryption There are 3 constraints here: Full TLS handshake has to be captured CheckAppendix 2to learn how to to disable BIG-IP's cache RSA key exchange has to be used, i.e. no (EC)DHE CheckAppendix1to understand how to check what's key exchange method used in your TLS connection CheckAppendix 2to understandhow to prioritise RSA as key exchange method Private key has to be copied to Wireshark machine (ssldump command solves this problem) Roughly, to accomplish that we can setCache Sizeto 0 on SSL profile and remove (EC)DHE fromCipher Suites(seeAppendix 1for details) I first took a packet capture using :pmodifier to capture only the client and server flows specific to my Client's IP address (10.199.3.1): Note: The0.0interface will capture any forwarding plane traffic (tmm) andnnnis the highest noise to capture as much flow information as possible to be displayed on the F5 dissector header.For more details about tcpdump syntax, please have a look atK13637: Capturing internal TMM information with tcpdumpandK411: Overview of packet tracing with the tcpdump utility. Also,we need to make sure we capture the full TLS handshake. It's perfectly fine to captureresumed TLS sessionsas long as full TLS handshake has been previously captured. Initially, our capture is unencrypted as seen below: On Mac, I clicked onWireshark→Preferences: ThenProtocols→TLS→RSA keys listwhere we see a window where we can reference BIG-IP's (or server if we want to decrypt server SSL side) private key: Once we get there, we need to add any IP address of the flow we want Wireshark to decrypt, the corresponding port and the private key file (default.crtfor Client SSL profile in this particular lab test): Note:For Client SSL profile, this would be the private key onCertificate Chainfield corresponding to the end entity Certificate being served to client machines through the Virtual Server. For Server SSL profile, the private key is located on the back-end server and the key would be the one corresponding to the end entity Certificate sent in the TLSCertificatemessage sent from back-end server to BIG-IP during TLS handshake. Once we clickOK, we can see the HTTP decrypted traffic (in green): In production environment, we would normally avoid copying private keys to different machines soanother option is usessldumpcommand directly on the server we're trying to capture. Again, if we're capturing Client SSL traffic,ssldumpis already installed on BIG-IP. We would follow the same steps as before but instead of copying private key to Wireshark machine, we would simply issue this command on the BIG-IP (or back-end server if it's Server SSL traffic): Syntax:ssldump-r<capture.pcap>-k<private key.key>-M<type a name for your ssldump file here.pms>. For more details, please have a look atK10209: Overview of packet tracing with the ssldump utility. Inssldump-generated.pms, we should find enough information for Wireshark to decrypt the capture: Syntax:ssldump-r<capture.pcap>-k<private key.key>-M<type a name for your ssldump file here.pms>. For more details, please have a look atK10209: Overview of packet tracing with the ssldump utility. Inssldump-generated.pms, we should find enough information for Wireshark to decrypt the capture: After I clickedOK, we indeed see the decrypted http traffic back again: We didn't have to copy BIG-IP's private key to Wireshark machine here. iRules The only constraint here is that we should apply the iRule to the virtual server in question. Sometimes that's not desirable, especially when we're troubleshooting an issue where we want the configuration to be unchanged. Note: there is abugthat affects versions 11.6.x and 12.x that was fixed on 13.x. It records the wrong TLS Session ID to LTM logs. The workaround would be to manually copy the Session ID from tcpdump capture or to use RSA decryption as in previous example. You can also combine bothSSL::clientrandomandSSL::sessionidwhich isthe ideal: Reference: K12783074: Decrypting SSL traffic using the SSL::sessionsecret iRules command (12.x and later) Again, I took a capture usingtcpdumpcommand: After applying above iRule to our HTTPS virtual server and taking tcpdump capture, I see this on /var/log/ltm: To copy this to a *.pms file wecanuseon Wireshark we can use sed command (reference:K12783074): Note:If you don't want to overwrite completely the PMS file make sure youuse >> instead of >. The endresult would be something like this: As both resumed and full TLS sessions have client random value, I only had to copy CLIENT_RANDOM + Master secret to our PMS file because all Wireshark needs is a session reference to apply master secret. To decrypt file on Wireshark just go toWireshark→Preferences→Protocols→TLS→Pre-Master Key logfile namelike we did inssldumpsection and add file we just created: As seen on LTM logs,CLIENTSSL_HANDSHAKEevent captured master secret from our client-side connection andSERVERSSL_HANDSHAKEfrom server side. In this case, we should have both client and server sides decrypted, even though we never had access to back-end server: Notice added anhttpfilter to show you both client and and server traffic were decrypted this time. tcpdump --f5 ssl option This was introduced in 15.x and we don't need to change virtual server configuration by adding iRules. The only thing we need to do is to enabletcpdump.sslproviderdb variable which is disabled by default: After that, when we take tcpdump capture, we just need to add --f5 ssl to the command like this: Notice that we've got a warning message because Master Secret will be copied to tcpdump capture itself, so we need to be careful with who we share such capture with. I had to update my Wireshark to v3.2.+ and clicked onAnalyze→Enabled Protocols: And enable F5 TLS dissector: Once we open the capture, we can find all the information you need to create our PMS file embedded in the capture: Very cool, isn't it? We can then copy the Master Secretand Client Random values by right clicking like this: And then paste it to a blank PMS file. I first pasted the Client Random value followed by Master Secret value like this: Note: I manually typedCLIENT_RANDOMand then pasted both values for both client and server sides directly from tcpdump capture. The last step was to go toWireshark→Preferences→Protocols→TLSand add it toPre-master-Secret log filenameand clickOK: Fair enough! Capture decrypted on both client and server sides: I usedhttpfilter to display only decrypted HTTP packets just like in iRule section. Appendix 1 How do we know which Key Exchange method is being used? RSA private key can only decrypt traffic on Wireshark if RSA is the key exchange method negotiated during TLS handshake. Client side will tell the Server side which ciphers it support and server side will reply with the chosen cipher onServer Hellomessage. With that in mind, on Wireshark, we'd click onServer Helloheader underCipher Suite: Key Exchange and Authentication both come before theWITHkeyword. In above example, because there's only RSA we can say that RSA is used for both Key Exchange and Authentication. In the following example, ECDHE is used for key exchange and RSAfor authentication: Appendix 2 Disabling Session Resumption and Prioritising RSA key exchange We can set Cache Size to 0 to disableTLS session resumptionand change the Cipher Suites to anything that makes BIG-IP pick RSA for Key Exchange:
TLS Fingerprinting - a method for identifying a TLS client without decrypting
Hello, Kevin Stewart here. A while back someone asked an interesting question in the DevCentral forum about selecting a client SSL profile based on the device (ex. iOS, Android, Windows Phone). Normally you'd use a browser User-Agent HTTP header to identify the client user agent, but in this case, and based on the OSI model, you wouldn't be able to select an SSL profile (OSI layer 6) based on a User-Agent HTTP header (OSI layer 7), because at this point in time you don't yet have the layer 7 data - it's still encrypted. You could, however, use layer 3 or 4 data (IPs and ports), but that's generally not useful for identifying the client user agent. But there might still be a way... Lee Brotherston has discovered that during an SSL handshake, most client user agents (different browsers, Dropbox, Skype, etc.) will initiate an SSL handshake request in an ever-so-unique way. The ordered combination of TLS version, record TLS version, ciphersuites, compression options, list of extensions, elliptic curves and signature algorithms are all specific enough that you can actually build a signature based on that data, and the collection of signatures into a database. From that discovery Lee created a project called "tls-fingerprinting". Please check it out. Now certainly, a client's ClientHello could be modified to support different ciphersuites and other features, the same way you could spoof a User-Agent HTTP header. However this modification will often lower security by re-introducing previously unsupported options, or in many cases modification to the user agent's SSL parameters isn't easy or isn't possible. So given that we now have a new way to identify a user agent based on the client's ClientHello (the first message in the SSL handshake), I decided to re-visit the original DC forum request by integrating Lee's database into an iRules-based solution. The code example in the aforementioned thread just used the client's ciphersuite list, however, so today I'm going to expand on that and use all of the paramaters from the tls-fingerprinting database. Before we get started, I should mention two things: This article is going to be long (sorry about that), and We're going to break it down into a few phases, specifically Defining the values in the tls-fingerprinting signature Exporting and converting the tls-fingerprinting database to a BIG-IP external data group Creating a fingerprintTLS PROC iRule (name this "Library-Rule") Creating the caller iRule So let's get started. Defining the values in the tls-fingerprinting signature Here's an example signature entry in Lee's tls-fingerprinting database (JSON version): {"id": 0, "desc": "ThunderBird (v38.0.1 OS X)", "record_tls_version": "0x0301", "tls_version": "0x0303", "ciphersuite_length": "0x0016", "ciphersuite": "0xC02B 0xC02F 0xC00A 0xC009 0xC013 0xC014 0x0033 0x0039 0x002F 0x0035 0x000A", "compression_length": "1", "compression": "0x00", "extensions": "0x0000 0xFF01 0x000A 0x000B 0x0023 0x0005 0x000D 0x0015" , "e_curves": "0x0017 0x0018 0x0019" , "sig_alg": "0x0401 0x0501 0x0201 0x0403 0x0503 0x0203 0x0402 0x0202" , "ec_point_fmt": "0x00" } Broken down it looks like this: { "id": 0, "desc": "ThunderBird (v38.0.1 OS X)", "record_tls_version": "0x0301", "tls_version": "0x0303", "ciphersuite_length": "0x0016", "ciphersuite": "0xC02B 0xC02F 0xC00A 0xC009 0xC013 0xC014 0x0033 0x0039 0x002F 0x0035 0x000A", "compression_length": "1", "compression": "0x00", "extensions": "0x0000 0xFF01 0x000A 0x000B 0x0023 0x0005 0x000D 0x0015" , "e_curves": "0x0017 0x0018 0x0019" , "sig_alg": "0x0401 0x0501 0x0201 0x0403 0x0503 0x0203 0x0402 0x0202" , "ec_point_fmt": "0x00" } This is a pretty straight forward set of JSON key-value pairs. And if you're curious about what any of these values mean, I urge you to fire up Wireshark, open a browser to some HTTPS site, and then find a ClientHello message in the capture. You'll see all of these values and more, except for the first two, in that message. Our job then is to a) export the set of signatures to a BIG-IP external data group, and b) create an iRule that extracts all of these values from the client's ClientHello and compares those to the set of signatures in the data group. Of course iRules don't natively support JSON parsing, and while yes I could use iRulesLX for this, I decided to simply reformat the signatures in the data group to something more condusive to TCL iRules. Exporting and converting the tls-fingerprinting database to a BIG-IP external data group I don't really care about the "id" value, so I'll leave that out. And the "desc" field will be the value in the data group. The key will be the concatenation of all of the remaining fields. "signature_data" := "ThunderBird (v38.0.1 OS X)", I'm also going to remove the "0x" from the hex values, remove whitespace, and delimit each field with the plus (+) sign, so the resulting key for the above signature will look like this: 0301+0303+0016+C02BC02FC00AC009C013C01400330039002F0035000A+1+00+0000FF01000A000B00230005000D0015+001700180019+04010501020104030503020304020202+00 If any of the values don't exist (ex. the signature doesn't have a sig_alg value), that value is replaced with "@@@@" in the resulting key. Okay, so to convert the tls-fingerprinting database to a BIG-IP external data group, you have to: 1. Download it - the current JSON-based database is here: https://github.com/LeeBrotherston/tls-fingerprinting/blob/master/fingerprints/fingerprints.json. Copy that JSON data in whole to a local text file. The conversion script uses BASH, so you need a Linux or Mac box. I named the file 'fingerprint.db'. 2. Convert it - use the following BASH script to extract each of the fields from each of the signatures. I should warn you now that my sed/awk/grep foo isn't strong, so I borrowed a BASH JSON parser from here: https://gist.github.com/cjus/1047794 #!/bin/bash function jsonval () { ## description desc=`echo $1 | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w "desc" |awk -F": " '{print $2}'` ## record_tls_version rect=`echo $1 | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w "record_tls_version" |awk -F": " '{print $2}' |sed 's/0x//g'` if [ -z "$rect" ]; then rect="@@@@"; fi ## tls_version tlsv=`echo $1 | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w "tls_version" |awk -F": " '{print $2}' |sed 's/0x//g'` if [ -z "$tlsv" ]; then tlsv="@@@@"; fi ## ciphersuite_length cipl=`echo $1 | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w "ciphersuite_length" |awk -F": " '{print $2}' |sed 's/0x//g' |sed 's/ //g'` if [ -z "$cipl" ]; then cipl="@@@@"; fi ## ciphersuite ciph=`echo $1 | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w "ciphersuite" |awk -F": " '{print $2}' |sed 's/0x//g' |sed 's/ //g'` if [ -z "$ciph" ]; then tlsv="ciph"; fi ## compression_length coml=`echo $1 | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w "compression_length" |awk -F": " '{print $2}'` if [ -z "$coml" ]; then coml="@@@@"; fi ## compression comp=`echo $1 | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w "compression" |awk -F": " '{print $2}' |sed 's/0x//g' |sed 's/ //g'` if [ -z "$comp" ]; then comp="@@@@"; fi ## extensions exte=`echo $1 | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w "extensions" |awk -F": " '{print $2}' |sed 's/0x//g' |sed 's/ //g'` if [ -z "$exte" ]; then exte="@@@@"; fi ## e_curves ecur=`echo $1 | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w "e_curves" |awk -F": " '{print $2}' |sed 's/0x//g' |sed 's/ //g'` if [ -z "$ecur" ]; then ecur="@@@@"; fi ## sig_alg siga=`echo $1 | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w "sig_alg" |awk -F": " '{print $2}' |sed 's/0x//g' |sed 's/ //g'` if [ -z "$siga" ]; then siga="@@@@"; fi ## ec_point_fmt ecfp=`echo $1 | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w "ec_point_fmt" |awk -F": " '{print $2}' |sed 's/0x//g' |sed 's/ //g'` if [ -z "$ecfp" ]; then ecfp="@@@@"; fi echo "\"$rect+$tlsv+$cipl+$ciph+$coml+$comp+$exte+$ecur+$siga+$ecfp\" := \"$desc\"," } IFS=} for i in `cat fingerprint.db`; do jsonval $i done Create this BASH script however you like (VI, VIM, Joe, Nano, whatever), save it, chmod it so that it'll execute ('chmod 755 parser.sh'), and then run it ('./parser.sh'). It'll just echo the reformatted signatures to the screen, so you'll want to capture that as a file ('./parser.sh > fingerprint.dg'). It's also be a little slow, again due to my aggregious lack of sed/awk/grep (and regex) foo, but it should still finish in less than a minute. 3. Import it - now go to the BIG-IP management UI, and under System - File Management - Data Group File List, click Import. Choose your reformatted text file, give it a meaningful name (ex. fingerprint_db), select "String" as the File Contents type, and use the same name (ex. fingerprint_db) in the Data Group Name field. On a v12 BIG-IP this will auto-create the local data group. On earlier systems you'll need to go manually create the local data group object that points to this external data group. Creating a fingerprintTLS PROC iRule So now that we've reformatted and imported the fingerprintTLS database, let's build the iRule to parse out the data from the client's ClientHello. I should also warn you that this process requires a lot of binary manipulation, so please don't try to ingest it all at once if you're new to iRules. I'm building this iRule as a separate PROC that other data plane iRules can call. It won't be directly attached to a virtual server. ## Library-Rule ## TLS Fingerprint Procedure ################# ## ## Author: Kevin Stewart, 12/2016 ## Derived from Lee Brotherston's "tls-fingerprinting" project @ https://github.com/LeeBrotherston/tls-fingerprinting ## Purpose: to identify the user agent based on unique characteristics of the TLS ClientHello message ## Input: ## Full TCP payload collected in CLIENT_DATA event of a TLS handshake ClientHello message ## Record length (rlen) ## TLS outer version (outer) ## TLS inner version (inner) ## Client IP ## Server IP ############################################## proc fingerprintTLS { payload rlen outer inner clientip serverip } { ## The first 43 bytes of a ClientHello message are the record type, TLS versions, some length values and the ## handshake type. We should already know this stuff from the calling iRule. We're also going to be walking the ## packet, so the field_offset variable will be used to track where we are. set field_offset 43 ## The first value in the payload after the offset is the session ID, which may be empty. Grab the session ID length ## value and move the field_offset variable that many bytes forward to skip it. binary scan ${payload} @${field_offset}c sessID_len set field_offset [expr {${field_offset} + 1 + ${sessID_len}}] ## The next value in the payload is the ciphersuite list length (how big the ciphersuite list is. We need the binary ## and hex values of this data. binary scan ${payload} @${field_offset}S cipherList_len binary scan ${payload} @${field_offset}H4 cipherList_len_hex set cipherList_len_hex_text ${cipherList_len_hex} ## Now that we have the ciphersuite list length, let's offset the field_offset variable to skip over the length (2) bytes ## and go get the ciphersuite list. Multiple by 2 to get the number of appropriate hex characters. set field_offset [expr {${field_offset} + 2}] set cipherList_len_hex [expr {${cipherList_len} * 2}] binary scan ${payload} @${field_offset}H${cipherList_len_hex} cipherlist ## Next is the compression method length and compression method. First move field_offset to skip past the ciphersuite ## list, then grab the compression method length. Then move field_offset past the length (2) bytes and grab the ## compression method value. Finally, move field_offset past the compression method bytes. set field_offset [expr {${field_offset} + ${cipherList_len}}] binary scan ${payload} @${field_offset}c compression_len #set field_offset [expr {${field_offset} + ${compression_len}}] set field_offset [expr {${field_offset} + 1}] binary scan ${payload} @${field_offset}H[expr {${compression_len} * 2}] compression_type set field_offset [expr {${field_offset} + ${compression_len}}] ## We should be in the extensions section now, so we're going to just run through the remaining data and ## pick out the extensions as we go. But first let's make sure there's more record data left, based on ## the current field_offset vs. rlen. if { [expr {${field_offset} < ${rlen}}] } { ## There's extension data, so let's go get it. Skip the first 2 bytes that are the extensions length set field_offset [expr {${field_offset} + 2}] ## Make a variable to store the extension types we find set extensions_list "" ## Pad rlen by 1 byte set rlen [expr ${rlen} + 1] while { [expr {${field_offset} <= ${rlen}}] } { ## Grab the first 2 bytes to determine the extension type binary scan ${payload} @${field_offset}H4 ext ## Store the extension in the extensions_list variable append extensions_list ${ext} ## Increment field_offset past the 2 bytes of the extension type set field_offset [expr {${field_offset} + 2}] ## Grab the 2 bytes of extension lenth binary scan ${payload} @${field_offset}S ext_len ## Increment field_offset past the 2 bytes of the extension length set field_offset [expr {${field_offset} + 2}] ## Look for specific extension types in case these need to increment the field_offset (and because we need their values) switch $ext { "000b" { ## ec_point_format - there's another 1 byte after length ## Grab the extension data binary scan ${payload} @[expr {${field_offset} + 1}]H[expr {(${ext_len} - 1) * 2}] ext_data set ec_point_format ${ext_data} } "000a" { ## elliptic_curves - there's another 2 bytes after length ## Grab the extension data binary scan ${payload} @[expr {${field_offset} + 2}]H[expr {(${ext_len} - 2) * 2}] ext_data set elliptic_curves ${ext_data} } "000d" { ## sig_alg - there's another 2 bytes after length ## Grab the extension data binary scan ${payload} @[expr {${field_offset} + 2}]H[expr {(${ext_len} - 2) * 2}] ext_data set sig_alg ${ext_data} } default { ## Grab the otherwise unknown extension data binary scan ${payload} @${field_offset}H[expr {${ext_len} * 2}] ext_data } } ## Increment the field_offset past the extension data length. Repeat this loop until we reach rlen (the end of the payload) set field_offset [expr {${field_offset} + ${ext_len}}] } } ## Now let's compile all of that data. set cipl [string toupper ${cipherList_len_hex_text}] set ciph [string toupper ${cipherlist}] set coml ${compression_len} set comp [string toupper ${compression_type}] if { ( [info exists extensions_list] ) and ( ${extensions_list} ne "" ) } { set exte [string toupper ${extensions_list}] } else { set exte "@@@@" } if { ( [info exists elliptic_curves] ) and ( ${elliptic_curves} ne "" ) } { set ecur [string toupper ${elliptic_curves}] } else { set ecur "@@@@" } if { ( [info exists sig_alg] ) and ( ${sig_alg} ne "" ) } { set siga [string toupper ${sig_alg}] } else { set siga "@@@@" } if { ( [info exists ec_point_format] ) and ( ${ec_point_format} ne "" ) } { set ecfp [string toupper ${ec_point_format}] } else { set ecfp "@@@@" } ## Initialize the match variable set match "" ## Now let's build the fingerprint string and search the database set fingerprint_str "${outer}+${inner}+${cipl}+${ciph}+${coml}+${comp}+${exte}+${ecur}+${siga}+${ecfp}" ## Un-comment this line to display the fingerprint string in the LTM log for troubleshooting ## log local0. "${clientip}-${serverip}: fingerprint_str = ${fingerprint_str}" if { [class match ${fingerprint_str} equals fingerprint_db] } { ## Direct match set match [class match -value ${fingerprint_str} equals fingerprint_db] } elseif { not ( ${ciph} starts_with "C0" ) and not ( ${ciph} starts_with "00" ) } { ## Hmm.. there's no direct match, which could either mean a database entry doesn't exist, or Chrome (and Opera) are adding ## special values to the cipherlist, extensions list and elliptic curves list. ## ex. 9A9A, 5A5A, EAEA, BABA, etc. at the beginning of the cipherlist ## Let's strip out these anomalous values and try the match again. ## Substract 2 bytes from cipherlist length set cipl [format %04x [expr [expr 0x${cipl}] - 2]] ## Subtract 2 bytes from the front of the cipher list set ciph [string range ${ciph} 4 end] ## Subtract 2 bytes from the front of the extensions list set exte [string range ${exte} 4 end] ## There might be an additional random set in the string that needs to be removed (pattern is "(.)A\1A") regsub {(.)A\1A} ${exte} "" exte ## If the above regsub doesn't work, try the following: #regsub {(\wA)\1} ${exte} "" exte ## Subtract 2 bytes from the front of the elliptic curves list set ecur [string range ${ecur} 4 end] ## Rebuild the fingerprint string set fingerprint_str "${outer}+${inner}+${cipl}+${ciph}+${coml}+${comp}+${exte}+${ecur}+${siga}+${ecfp}" if { [class match ${fingerprint_str} equals fingerprint_db] } { ## Guess match set match [class match -value ${fingerprint_str} equals fingerprint_db] } else { ## No match set match "" } } ## Return the matching user agent string return ${match} } The PROC requires as input the full TCP payload (of the ClientHello message), the record length (extracted from the ClientHello message), the "outer" record TLS version and "inner" TLS version (also extracted form the ClientHello message). Using these values the PROC basically walks the payload looking for each of the required values (ciphersuite length, ciphersuite list, compression length, compression list, extensions list, elliptic curves, signature algorithms, and ec point formats). If any value doesn't exist in the payload (ex. the ClientHello doesn't contain a Sig_Alg field), that value is replaced with "@@@@". Once all of the values are found, the fingerprint string is created and used to search the data group. If there's a match, the user agent string (ex. ThunderBird (v38.0.1 OS X)) is returned to the caller. While testing this I noticed that newer versions of Chrome and Opera added what looked like "markers" to the ciphersuite list, extensions list, and elliptic curves list (ex. 9A9A, 5A5A, EAEA, BABA - always some alphanumeric value, followed by 'A', and repeated.). A cursory search didn't explain what these are, so maybe someone will know and report back. In the meantime, I added a "guess" function that removed these markers and tried the data group search again. All of the desktop browser testing (including Chrome and Opera) did get an accurate match with either the direct or guessed fingerprint, so I'll leave that in there until I find a better way to handle the markers. Creating the caller iRule The only thing left to do is to create the caller iRule. This iRule only needs to detect an SSL/TLS ClientHello, and then pass that to the fingerprint PROC. This is just a stub iRule to show the proper implementation. Once you've determined a TCP packet is an SSL/TLS handshake ClientHello, call the PROC and then do something useful with the resulting user agent string, like switch the client SSL profile. when CLIENT_ACCEPTED { ## Collect the TCP payload TCP::collect } when CLIENT_DATA { ## Get the TLS packet type and versions if { ! [info exists rlen] } { binary scan [TCP::payload] cH4ScH6H4 rtype outer_sslver rlen hs_type rilen inner_sslver if { ( ${rtype} == 22 ) and ( ${hs_type} == 1 ) } { ## This is a TLS ClientHello message (22 = TLS handshake, 1 = ClientHello) ## Call the fingerprintTLS proc set fingerprint [call Library-Rule::fingerprintTLS [TCP::payload] ${rlen} ${outer_sslver} ${inner_sslver} [IP::client_addr] [IP::local_addr]] ### Do Something here ### log local0. "match = ${fingerprint}" ### Do Something here ### } } # Collect the rest of the record if necessary if { [TCP::payload length] < $rlen } { TCP::collect $rlen } ## Release the paylaod TCP::release } What happens if there's no match? Yes, there are some caveats... It's safe to say that the tls-fingerprinting database isn't all inclusive. In fact it's FAR FROM COMPLETE and not always exact.I found, for example, that my version of Dropbox on a Win7 box (v16.4.30) doesn't make a match. It's nearly impossible to have the signature for every unique user agent every created, and all of the variations and versions of that agent. But what the database does have is the signatures for most browsers, so at the very least it makes for a nice way to whitelist browsers (vs. other agents). It also doesn't technically resolve the question in the original DC forum thread. The question was how to identify the device (ie. iOS, Android, Windows Phone), and for that you'd need some specific agent loaded on the device (not a browser) that could report that information. Mobile device management (MDM) solutions are particularly good at that sort of thing. The browser, Dropbox or other user agent on the mobile device may not specifically report the device (ex. "for iOS"). Some do, but I've found that most don't. At the end of this aticle I've included a few signatures that I found in my testing that aren't in the database. If your curious, uncomment thelog local0. "${clientip}-${serverip}: fingerprint_str = ${fingerprint_str}"line in the fingerprintTLS PROC and then tail the LTM log. The caller iRule is already logging the returned user agent string, so if that is empty, you'll see the empty match in the log (match =), preceded by the unmatched signature. Where this project may be most useful is in outbound traffic management, where you want to decrypt and inspect the Internet-bound traffic, but cannot decrypt some user agents becuase of things like cerificate pinning. Since the pinning decision happens at the client, the only other recourse is to bypass decryption and inspection based on the destination host name or IP address, which can be a tedious thing to manage. TLS fingerprinting might allow you to simply decrypt and inspect for the user agents that you know aren't affected by pinning, specifically browsers. You'll potentially miss some things that you could have decrypted, but you'll save yourself the burden of managing an ever-growing list of pinner exclusions. And on a final note, binary iRule manipulation is a very CPU-intensive thing to do. I could have very simply converted the raw payload to one long hex string (once) and walked that with string tools. I'll update the code when I have some time. Thanks. - Kevin Additional Signatures "0301+0303+0028+C02BC02F009ECC14CC13C00AC009C013C014C007C011003300320039009C002F0035000A00050004+1+00+0000FF01000A000B0023755000050012000D+001700180019+04010501020104030503020304020202+00" := "Dropbox", "0301+0303+0028+C02BC02F009ECC14CC13C00AC009C013C014C007C011003300320039009C002F0035000A00050004+1+00+0000FF01000A000B002300050012000D+001700180019+04010501020104030503020304020202+00" := "Dropbox", "0301+0303+001A+C030C028C014C02FC027C013009F006B0039009E0067003300FF+1+00+0000000B000A0023000D+00170019001C001B0018001A0016000E000D000B000C0009000A+060106020603050105020503040104020403030103020303020102020203+000102" := "Dropbox", "0301+0303+0094+C030C02CC032C02EC02FC02BC031C02D00A500A300A1009F00A400A200A0009EC028C024C014C00AC02AC026C00FC005006B006A006900680039003800370036C027C023C013C009C029C025C00EC00400670040003F003E0033003200310030C012C008C00DC00300880087008600850045004400430042001600130010000D009D009C003D0035003C002F00840041000A00FF+1+00+000B000A0023000D0015+00170019001C001B0018001A0016000E000D000B000C0009000A+060106020603050105020503040104020403030103020303020102020203+000102" := "Dropbox", "0301+0303+0028+C02BC02CC02FC030009E009FC009C00AC013C01400330039C007C011009C009D002F0035000500FF+1+00+0000000B000A0023000D+000E000D0019000B000C00180009000A00160017000800060007001400150004000500120013000100020003000F00100011+060106020603050105020503040104020403030103020303020102020203+000102" := "Android Google API Access", "0301+0303+001E+CC14CC13C02BC02F009EC00AC0140039C009C0130033009C0035002F000A+1+00+FF01000000170023000D0005337400120010000B000A+00170018+0601060305010503040104030301030302010203+00" := "Chrome 47.0.2526.83", "0301+0303+001E+CC14CC13C02BC02F009EC00AC0140039C009C0130033009C0035002F000A+1+00+FF01000000170023000D0005337400120010000B000A+00170018+0601060305010503040104030301030302010203+00" := "Chrome 48.0.2564.97", "0301+0303+001E+CC14CC13C02BC02F009EC00AC0140039C009C0130033009C0035002F000A+1+00+FF01000000170023000D00053374001200107550000B000A0015+00170018+0601060305010503040104030301030302010203+00" := "Chrome 48.0.2564.97", "0301+0303+0022+CCA9CCA8CC14CC13C02BC02FC02CC030C009C013C00AC014009C009D002F0035000A+1+00+FF01000000170023000D0005001200107550000B000A0018+001D00170018+06010603050105030401040302010203+00" := "Android Silk Browser", "0301+0303+0022+CCA9CCA8CC14CC13C02BC02FC02CC030C009C013C00AC014009C009D002F0035000A+1+00+FF01000000170023000D0005001200107550000B000A00180015+001D00170018+06010603050105030401040302010203+00" := "Android Silk Browser" "0303+0303+0038+C02CC02BC030C02F009F009EC024C023C028C027C00AC009C014C01300390033009D009C003D003C0035002F000A006A0040003800320013+1+00+0005000A000B000D0023001000175500FF01+001D00170018+040105010201040305030203020206010603+00" := "Internet Explorer 11.447.14393.0(Win 10)", "0301+0303+0022+C02BC02FC02CC030CCA9CCA8CC14CC13C009C013C00AC014009C009D002F0035000A+1+00+FF0100170023000D0005001200107550000B000A+001D00170018+06010603050105030401040302010203+00" := "Chrome 55.0.2883.87",TLS Stateful vs Stateless Session Resumption
1. Preliminary Information TLS Session Resumption allows caching of TLS session information. There are 2 kinds: stateful and stateless. In stateful session resumption, BIG-IP stores TLS session information locally. In stateless session resumption, such job is delegated to the client. BIG-IP supports both stateful and stateless TLS session resumption. Enabling stateful or stateless session resumption is just a matter of ticking/unticking a tickbox on LTM's Client SSL profile: In this article, I'm going to walk through how session resumption works by performing a lab test. Here's my topology: Do not confuseSession Reuse/Resumption withRenegotiation. Renegotiation uses the same TCP connection to renegotiate security parameters which does not involved Session ID or Session Tickets. For more information please refer to SSL Legacy Renegotiation Secure Renegotiation. 2. How Stateful Session Resumption works Capture used:ssl-sample-session-ticket-disabled.pcap 2.1 New session Statefulmeans BIG-IP will keep storing session information from as many clients its cache allows and TLS handshake will proceed as follows: We can see above that Client sends an empty Session ID field and BIG-IP replies with a new Session ID (filter used:tcp.stream eq 0). After that, full handshake proceeds normally where Certificate and Client Key Exchange are sent and there is also the additional cost CPU-wise to compute the keys: 2.2 Reusing Previous Session Now both Client and Server have Session ID 56bcf9f6ea40ac1bbf05ff7fd209d423da9f96404103226c7f927ad7a2992433 stored in their TLS session cache. The good thing about it is that in the next TLS connection request, client won't need to go through the full TLS handshake again. Here's what we see: Client just sends Session ID (56bcf9f6ea40ac1bbf05ff7fd209d423da9f96404103226c7f927ad7a2992433) it previously learnt from BIG-IP (via Server Hello from previous connection) on its Client Hello message. BIG-IP then confirms this session ID is in its SSL Session cache and they both go through what is known as abbreviated TLS handshake. No certificate or key information is exchanged during abbreviated TLS handshake and previously negotiated keys are re-used. 3. How Stateless Session Resumption works Capture used:ssl-sample-session-ticket-enabled-2.pcap 3.1 New Session Because of the burden on BIG-IP that has to store one session per client,RFC5077suggested a new way of doing session Resumption that offloads the burden of keeping all TLS session information to client and nothing else needs to be stored on BIG-IP. Let's see the magic! Client first signals it supports stateless session resumption by adding SessionTickets TLS extension to its Client Hello message (in green): BIG-IP also signals back to client (in red) it supports SessionTicket TLS by adding empty SessionTicket TLS extension.Notice thatSession IDis NOT used here! The TLS handshake proceeds normally just like in stateful session resumption. However, just before handshake is completed (with Finished message), BIG-IP sends a new TLS message calledNew Session Ticketwhich consists of encrypted session information (e.g. master secret, cipher used, etc) where BIG-IP is able to decrypt later using a unique key it generates only for this purpose: From this point on, client (10.199.3.135) keeps session ticket in its TLS cache until next time it needs to connect to the same server (assuming session ticket did not expire). 3.2 Reusing Previous Session Now, when the same client wants to re-use previous session,it forwards the same session ticket aboveinSessionTicket TLS extensionon its Client Hello message as seen below: As we've noticed, Client also creates a new Session ID used for the following purpose: Server replies back with same session ID: BIG-IP accepted Session Ticket and is going to reuse the session. Server replies with empty/different Session ID: BIG-IP decided to go through full handshake either because Session Ticket expired or it is falling back to stateful session resumption. PS: Such session ID is NOT stored on BIG-IP otherwise it would defeat the purpose of stateless session reuse. It is a one-off usage just to confirm to client BIG-IP accepted session ticket they sent and we're not going to generate new session keys. In our example, BIG-IP successfully accepted and reused TLS session. We can confirm that an abbreviated TLS handshake took place and on Server Hello message BIG-IP replied back with same session ID client sent (to BIG-IP): Now, client This is session resumption in action and BIG-IP doesn't even have to store session information locally, making it a more scalable option when compared to stateful session resumption.How Proxy SSL works on BIG-IP
1. Lab Scenario Lab test results: Client completes 3-way handshake with BIG-IP and BIG-IP immediately opens and completes 3-way handshake with back-end server Upon receiving Client Hello on client-side, BIG-IP immediately sends Client Hello on server-side as it is BIG-IP copies same cipher suite list seen on client-side Client Hello to server-side Client Hello BIG-IP ignores what is configured on both Client SSL or Server SSL profiles. As soon as BIG-IP receives Server Hello it confirms two things: Is RSA the chosen key exchange mechanism in Cipher Suite on Server Hello message sent from back-end server? Does Certificate sent by back-end matches the one configured on Server SSL profile on BIG-IP? ONLY if both conditions above match BIG-IP proceeds with handshake. Otherwise, BIG-IP sends a Handshake Failure (Fatal Alert) to both Server and Client with reset cause ofillegal_parameter BIG-IP copies same Server SSL/Back-end Server certificate to Certificate message sent to Client on client-side BIG-IP completely ignores certificate you configured on Client SSL. It always uses the same server-side certificate. Assuming TLS handshake completes successfully BIG-IP is able to decrypt all client-side as well as server-side data which is the whole purpose of Proxy SSL. 2. How Proxy SSL works When Proxy SSL is enabled,BIG-IP does its best to match client-side to server-side connection in terms of negotiation and traffic to make it as transparent as possible to both client and back-end server and at the same allowing BIG-IP to decrypt traffic. About data transparency BIG-IP achieves transparency by copying whatever client and server sends back and forth. About data decryption BIG-IP has an extra configuration requirement for Proxy SSL configuration (according toK13385) that you should add the same certificate/key present on the back-end server to Certificate/Key fields on Server SSL proxy of BIG-IP. This way BIG-IP can decrypt both client and server sides of connection. In practical terms and to achieve this, BIG-IP completely changes the original purpose of Server SSL Certificate and Key fields. Here's my config: Certificate/Key fields above are no longer used for the purpose described inK14806, i.e. to independently authenticate BIG-IP to back-end server through Client Authentication. Instead, when Proxy SSL is enabled it is used to validate if Certificate sent by back-end server is the same one in this field. If so, BIG-IP also copies this certificate to Certificate message sent to Client on client-side. Noticethat when Proxy SSL is NOT bypassedCertificate configured on Client SSL profile is never used. As soon as I sent first request to Wildcard forwarding VIP BIG-IP establishes TCP connection on client-side first and then immediately on server-side: The next step is to forward exactly the sameClient Hellowe receive from Client on client-side to Server on server-side: Now server sends Server Hello: The first thing BIG-IP checks is key exchange mechanism as Proxy SSL has to use RSA (frame 12): Now BIG-IP checks if Certificate insideCertificatemessage is the same as the one configured on Server SSL. In this case I have used Certificate's unique serial number to confirm this. On BIG-IP: Now on Server's message (frame 12): Now BIG-IP sends this same certificate inServer Hellomessage client-side and we can confirm from Serial number that it is the same: From this moment handshake should complete successfully and BIG-IP is maintaining 2 separate connections using the same certificate/key pair on client and server side with the ability to decrypt both. 3. Troubleshooting Proxy SSL when BIG-IP is the culprit Typically, if BIG-IP is the culprit it either because back-end server selected non-RSA key exchange cipher or because cert/key which are not supported. In another test I used DHE on purpose andBIG-IP resets connection immediately after Server Hello message is received from back-end server which is typical sign of validation error: I confirmed back-end server had selected ECDHE key exchange cipher which is not supported by Proxy SSL (frame 12): In case you don't know yet here's how you work out key exchange cipher: Disabling non-RSA cipher on back-end server did the trick to fix the above error as Proxy SSL only supports RSA key exchange. 4. Setting Up Proxy SSL on BIG-IP I used very minimal configuration for this lab and the only thing I did was to create a wildcard forwarding virtual server using Standard VIP: I enabled proxy-ssl on both Client and Server SSL profiles and added back-end server certificate (ltm3.crt) and key (ltm3.key) to Server SSL profile: I also disabled (EC)DHE and explicitly configured RSA as key exchange mechanism in my back-end server. I also confirmed back-end server was also usingltm3.crt/ltm3.keyas it must match the one configured on Server SSL profile: And it all worked fine:Understanding Server SSL Trusted Certificate Authorities on BIG-IP
1 Quick Intro This article is about how this Server SSL option works on BIG-IP: We'll create a certificate chain for 2 back-end servers using OpenSSL and copy the root CA of each server to Trusted Certificate Authorities on Server SSL profile: This way, when BIG-IP is configured to validate back-end server certificates, i.e. when Server Certificate (peer-cert-mode in tmsh) is set to require, BIG-IP can trust back-end certificates that were signed by certificates in the file we added to Trusted Certificate Authorities (ca-file in tmsh). Also, because Trusted Certificate Authorities file only contain the Root CAs (and not the intermediate ones), back-end servers should also send the intermediate certificates so that the whole chain can be verified by BIG-IP. Note: for production environment, you'll probably going to use the same chain on both servers. The reason I create 2 separate chains here was just to add 2 concatenated CA's public keys to Trusted Certificate Authorities for demonstration purposes. For the purpose of this article, here's what we're going to be doing if you follow along: Creating Certificate Chain using OpenSSL Adding Certificate Chain to Apache and NGINX Adding Root CAs to Trusted Certificate Authorities on BIG-IP Confirming it all works with curl command 2 Creating Certificate Chain using OpenSSL First we create the Root Certificate Authority that is going to sign our intermediate certificate. Note that the same steps are done for server2 bundle so I'm not repeating it here. I first created a directory to keep our certificate/key files in: The next steps would be to create the CAs and the end entity certificate along with corresponding keys. 2.1 Creating server1_rootCA.key The private key (server1_rootCA.key) is the first one to be created since public key is generated from private key: Note: Are you wondering why I added-aes256 which is symmetric key whilst creatinga private key, which is asymmetric key? That's only to protect the private key's contents in disk. For example, if an attacker gets hold of server1_rootCA.key, they can create a public key with such a key or decrypt any TLS traffic protected by server1_rootCA.crt. This is because our keys here are PEM encoded by default, but not encrypted. To avoid this, we can encrypt it at disk (known as encryption at rest) using symmetric key encryption with AES 256. By the way, openssl command to verify contents of private key is openssl rsa -text -in <key>.key 2.2 Creating server1_rootCA.crt Now we create our public key: Note: The required fields were filled for illustration purposes. 2.3 Creating server1_intermediateCA.key As always, we also create the private key first: 2.4 Creating server1_intermediateCA.csr Wow, wait! Why .csr? Why not .crt like we did before? The reason is because our intermediate CA will be signed by our Root CA, remember? We'd only create a .crt directly if we were creating a self-signed certificate like our Root CA. The idea of the .csr is that we'll set up all the information we want our certificate to have and we'll use our Root CA to turn it into a .crt file. Does it make sense? 2.5 Creating server1_intermediateCA.crt Based on the information from server1_intermediateCA.csr, we can now use server1_rootCA.key to create server1_intermediateCA.crt. However, before we do that, we may want to add some capabilities to our intermediate certificate which is recommended by man x509v3_config command: Note: the flags above are out of the scope of this article but setting basicConstraints to critical means that for validation purposes, the flags we're setting have to be taken into consideration. For example, CA:true means that this is a CA capable of signing other certificates such as our end-of-chain certificate that we'll create after that. We'd set this flag to false when creating end of chain client or server certificates as they're not supposed to sign other certificates. The keyUsage indicates what this CA should be used for. We're now using these extensions to create and sign server1_intermediateCA.crt using Root CA's private key: This way, if BIG-IP has Root CA's public key, it can confirm that this certificate was indeed signed by Root CA's private key. 2.6 Creating server1_endofchain.key Just like Root and Intermediate CAs, we create private key first: 2.6 Creating server1_endofchain.csr We now type the info we want our .crt file to have once it's signed by our intermediate CA: 2.6 Creating server1_endofchain.crt We also have an extension file here with the typical back-end server flags and constraints that can be found in man page as well: And we now create server1_intermediateCA.crt: We can also verify csr/crt/key as they should all output the same md5 hash: Note: For server2, we use the same steps but replacing server1 with server2 when appropriate. 3 Adding Certificate Chain to WebServer 3.1 Apache If not enabled, activate ssl module using a2enmod command: Don't restart Apache just now because we'll first add our end entity certificate and intermediate certificate to Apache's config file first. First I'll the default ssl file from sites-available to sites-enabled because only files in sites-enabled are actually active on the webserver. As Apache already has a default template, I'll create a symbolic link as this is just for demonstration purposes: And the only thing I did was to edit SSLCertificateFile, SSLCertificateKeyFile and SSLCertificateChainFile and add our end entity certificate, end entity key and intermediate CA certificate respectively: Once we try to restart Apache, we need to type in our passphrase for the certificates encrypted with AES256 at rest, remember? Otherwise, Apache won't be able to read it. We now confirm Apache is running properly: And most importantly, that we can reach our SSL protected website locally: And from BIG-IP: Note: the k flag on curl command skips certificate validation and for the moment that's what we want. Even if we add certificate validation to BIG-IP configuration, we need to make sure we understand that our curl command above is executed from BIG-IP's control plane (Linux) so a certificate added to BIG-IP's configuration file only applies to BIG-IP's forwarding plane (tmm). Therefore, even after we add the correct valid certificate to Trusted Certificate Authorities, the curl command above will not automatically reference it. 3.2 NGINX If we're using NGINX webserver, we should bundle our end entity certificate together with intermediate CA's certificate: And add it to your webserver's configuration on /etc/nginx/conf.d/ directory: In this case, NGINX will send 2 certificates (end entity and intermediate one) back to BIG-IP. FYI, a bundled chain is just one PEM certificate concatenated with the other: Not that hard to picture, is it? And lastly, we reload NGINX: For more details, please have a look at NGINX documentation. 4 Adding Root CAs to Trusted Certificate Authorities on BIG-IP 4.1 What do we add to Trusted Certificate Authorities? One important concept to grasp is that we don't need to copy the whole certificate chain to BIG-IP. The chain belongs to your back-end server. The only thing we need from the chain to add to BIG-IP is the top-level Root certificate(s) that signed the chain BIG-IP will verify. From the point of view of BIG-IP, we'll make it trust whichever certificate is signed by CA list added to Trusted Certificate Authorities. Therefore, we only need to add server1_rootCA.crt and server2_rootCA.crt in our lab test. I have copied both Root CAs from Server1 and Server2 to a temporary directory on BIG-IP: I'll make it a bundle named servers_rootCA.crt: 4.2 Importing Root CA to BIG-IP 4.2.1 Via GUI We can now import it to BIG-IP. In the GUI on v15.x, you'd go to Certificate Management → Traffic Certificate Management → SSL Certificate List → Import: Select Import Type → Certificate: And upload (or copy/paste it if you wish) the contents of servers_rootCA.crt: 4.2.2 Via TMSH As BIG-IP versions change, GUI is more susceptible to changes. If that is the case and this article ever gets outdated, here's the tmsh command equivalent to import certificate chain: 4.3 Adding Chain to Trusted Certificate Authorities If we go to our Server SSL profile we should now see servers_rootCA.crt as option to add to Trusted Certificate Authorities: And we just click on Update. Note: notice that Server Certificate is set to require! Don't forget that! 4.4 Final test with Curl command For this test I disabled Generic Alert on Server SSL profile: Otherwise, BIG-IP would not tell us the real reason why handshake failed. 4.4.1 Trusted Certificate Authorities set to None When I issued curl command from my client it failed as expected because Server Certificate is set to require and we added nothing to Trusted Certificate Authorities: And I took the opportunity to take a tcpdump capture in parallel: And on Wireshark we can clearly see the reason why our curl command failed: BIG-IP doesn't recognise the CA as trusted because it's not on Trusted Certificate Authorities. In fact, there's nothing there so any back-end certificate would fail anyway. 4.4.2 Trusted Certificate Authorities set to servers_rootCA.crt Now our curl command works: I also took a tcpdump capture here: In fact, one thing that is interesting to note is that we can see that Certificate message sent from back-end server only has the intermediate CA and the end entity certificate: If we wanted, we could've configured Apache or NGINX to add Root CA to the bundle but it's not necessary. The reason why is because BIG-IP already has the Root CA in the bundle we added to Trusted Certificate Authorities, remember? That's it for now. Appendix 1 When your certificate is signed by a well-known Root CA Web browsers have somewhere close to a thousand trusted Root CAs by default. These were added by the developers of your web browser. BIG-IP has a default file called ca-bundle.crt with the usual browser's trusted CA repository and on v15.x it contains 857 trusted CAs: If your certificate was signed by one of the major CAs out there, it's likely the only thing you'll need to do in production would be to use ca-bundle.crt: Why? Because if the Root CA that signed your certificate is already in ca-bundle.crt then there's no need to use a different bundle. It's up to you, really!Tools and facilities to troubleshoot HTTP/3 over QUIC with the BIG-IP system
Introduction This article is for engineers who are troubleshooting issues related to HTTP/3 over QUIC as you deploy this new technology on your BIG-IP system. As you perform your troubleshooting tasks, the BIG-IP system provides you a set of useful tools along with other third party software to identify the root cause of issues and even tune HTTP/3 performance to maximize your system's potential. Overview of HTTP/3 and QUIC HTTP/3 is the next version of the HTTP protocol after HTTP/2. The most significant change in HTTP/3 from its predecessors is that it uses the UDP protocol instead of TCP. HTTP/3 uses a new Internet transport protocol, QUIC uses streams at the transport layer and provides TCP-like congestion control and loss recovery. One major improvement QUIC provides is it combines the typical 3-way TCP handshake with TLS 1.3's handshake. This improves the time required to establish a connection. Hence, you may see QUIC as providing the functions previously provided by TCP, TLS, and HTTP/2 as shown in the following diagram: For an overview of HTTP/3 over QUIC with the BIG-IP system, refer to K60235402: Overview of the BIG-IP HTTP/3 and QUIC profiles. Available tools and facilties Beginning in BIG-IP 15.1.0.1, HTTP/3 over QUIC (client-side only) is available as an experimental feature on the BIG-IP system. Beginning in BIG-IP 16.1.0, BIG-IP supports QUIC and HTTP/3. In addition to that feature, there are tools and facilities that are available to help you troubleshoot issues you might encounter. Install an HTTP/3 command line client Use a browser that supports QUIC Review statistics on your BIG-IP system Enable QUIC debug logging on the BIG-IP system Perform advanced troubleshooting with Packet Tracing Use the NetLog feature from the Chromium Project to capture a NetLog dump. Use the tcpdump command and Wireshark to capture and analyze traffic. Use the qlog trace system database key on the BIG-IP system. Important: For BIG-IP versions prior to 16.1.0 that are in the experimental stages, it is important that you note in your troubleshooting, the version of the ietf draft that your client and server implements. For example, in the Hello packets between the client and server, version negotiation is performed to ensure that client and server agree to a QUIC version that is mutually supported. In BIG-IP 15.1.0.1, the HTTP/3 and QUIC profiles in the BIG-IP system are experimental implementations of draft-ietf-quic-http-24 and draft-ietf-quic-transport-24 respectively. You need to consider this when configuring the Alt-Svc header in HTTP/3 discovery. For some browsers such as those from the Chromium project, Chrome canary, Microsoft Edge canary and Opera, when starting these browsers from the command line, you need to provide the QUIC version it implements. For example, for Chrome canary, you run the following command: chrome.exe --enable-quic --quic-version=h3-25. Only implementations of the final, published RFC can identify themselves ash3. Until such an RFC exists, implementations must not identify themselves using theh3string. 1. Install an HTTP/3 command line client Keep in mind that HTTP/3 over QUIC runs on UDP instead of TCP. By default, browsers always initiate a connection to the server using the traditional TCP handshake which will not work with a QUIC server listening for UDP packets. You therefore need to configure HTTP/3 discovery on your BIG-IP system. This can be done by using the HTTP Alternative Services concept which can be implemented either by inserting the Alt-Svc header or via DNS as a HTTPSVC DNS resource record. To insert the Alt-Svc header, refer to K16240003: Configuring HTTP/3 discovery for BIG-IP virtual server. As you troubleshoot your HTTP/3 discovery implementation, you can use a command line tool that does not come with the overhead of HTTP/3 discovery. Following are two popular tools that you can install on your client system: The picoquic client The curl client where you have the option to use either the ntcp2 or quiche software libraries. 2. Use a browser that supports QUIC At this time, browsers by default, still do not support QUIC and do not send the server UDP packets to establish a QUIC connection. The following browsers which are in development support it: Firefox Nightly Chrome Canary (Chromium Project) Microsoft Edge Canary (Chromium Project) Opera (Chromium Project) Note that for browsers from the Chromium project, you need to specify the QUIC ietf version that the browser supports when you launch it. For example, for Chrome, run the following command: chrome.exe --enable-quic --quic-version=h3-25. In most browsers today, you can quickly view the HTTP information exchanged by using the built-in developer tool. To open the tool, select F12 after your browser opens and access any site that supports QUIC. Select the Network tab and under the Protocol column, look at h3-<draft_version> . If the Protocol column is not there, you may have to right click the toolbar to add it. Note: Only implementations of the final, published RFC can identify themselves as h3. Until such an RFC exists, implementations must not identify themselves using the h3 string. Click the name of the HTTP request and you can see that the site returns the Alt-Svc header indicating that it supports HTTP/3 with its ietf draft version. 3. Review statistics on your BIG-IP system The statistics facility on the BIG-IP system displays the system's QUIC traffic processing. On the Configuration utility, go to Local Traffic > Virtual Servers. Select the name of your virtual server and select the Statistics tab. In the Profiles section, select the HTTP/3 and QUIC profiles associated with the virtual server. Alternatively, you can view the statistics from the TMOS shell (tmsh) utility using the following command syntax: tmsh show ltm profile http3 <http3_profile_name> tmsh show ltm profile quic <quic_profile_name> 4. Enable QUIC debug logging on the BIG-IP system You can use the sys db variable tmm.quic.log.level to adjust the verbosity of the QUIC log level to the /var/log/ltm file. Type the following command to see the list of values. tmsh list sys db tmm.quic.log.level value-range sys db tmm.quic.log.level { default-value "Critical" value-range "Critical Error Warning Notice Info Debug" } For example: tmsh modify sys db tmm.quic.log.level value debug 5. Perform advanced troubleshooting with Packet Tracing a. Use the NetLog feature from the Chromium Project to capture a NetLog dump. NetLog is an event logging mechanism for Chrome’s network stack to help debug problems and analyze performance not just for HTTP/3 over QUIC traffic but also HTTP/1.1 and HTTP/2. This feature is available only in browsers from the Chromium project, such as Google Chrome, Opera and Microsoft Edge. The feature provides detailed client side logging including SSL handshake and HTTP content without having to perform decryption or run any commands on your BIG-IP system. To start capturing, open a new tab on your browser and go to, for example, chrome://net-export (Chromium only). For a step by step guide, refer to How to capture a NetLog dump. Once you have your NetLog dump, you can view and analyze it by navigating to netlog-viewer (Chromium only). To analyze QUIC traffic, on the left panel, select Events. In the Description column, identify the URL you requested. For QUIC SSL handshake events, select QUIC_SESSION. For HTTP content, select URL_REQUEST. For example, in the following NetLog dump, the connection failed at the beginning because the client and server could not negotiate a common QUIC version. b. Use the tcpdump command and Wireshark to capture and analyze traffic. The tcpdump command and Wireshark are both essential tools when you need to examine any communication at the packet level. To generate captures and the SSL secrets required to decrypt them, follow the procedure in K05822509: Decrypting HTTP/3 over QUIC with Wireshark. Keep your Wireshark version updated at all times as Wireshark's ability to decode QUIC packets continue to evolve as we speak. c. Use the qlog traces on the BIG-IP system. The BIG-IP qlog trace facility provides you another tool to troubleshoot QUIC communications. By enabling a database variable, the system logs packets and other events to /var/log/trace<TMM_number>.qlog files. qlog is a standardized structured logging format for QUIC and is basically a well-defined JSON file with specified event names and associated data that allows you to use tools like qvis for visualization. Note that the payload is not logged. The qlog trace files are compliant to the IETF schema specified in draft-marx-qlog-main-schema-01. To capture and analyze qlog trace files on your BIG-IP system, perform the following procedure: Capturing qlog trace files on the BIG-IP system Login to the BIG-IP command line. Enable qlogging by typing the following command: tmsh modify sys db quic.qlogging value enable Reproduce the issue you are troubleshooting by initiating QUIC traffic to your virtual server. Disable qlogging by typing the following command: tmsh modify sys db quic.qlogging value disable Note: This step will log required closing json content to the trace files, terminate the trace logging gracefully and is required before you view the files. Sanitizing the qlog trace files Before loading the trace files onto a graphical visualization tool, you first need to sanitize the json content. The tools attempt to fix some common json errors but there may be cases where you need to manually correct some json syntax errors by adding closing braces or commas. Note: Knowledge of different json constructs such as objects, arrays and members may be required when you fix the json files. You can use any of the available online tools such as the following: Json Formatter Fixjson freeformatter Important: Even as the payload information such as IP addresses, or HTTP content are not included in the trace files, you should exercise caution when uploading content to online tools. F5 is not responsible for the privacy and security of your data when you use the third party software listed in this procedure. Alternatively, you can download and install any of the following command line tools on your client device: jsonlint-php jsonlint-py jsonlint-cli Loading and analyzing the qlog trace files with a visualization tool When you have sanitized your json trace files, upload them to a visualization tool for analysis. For example, you can use the following tool available for free. qvis QUIC and HTTP/3 visualization toolsuite The visualization tool can provide you graphical representations of the sequence of messages, congestion information and qlog stats for troubleshooting. For example, the following screenshot, illustrates a sequence diagram of the SSL handshake. Important: Even as payload information such as IP addresses, or HTTP content are not included in the trace files, you should exercise caution when uploading content to online tools. F5 is not responsible for the privacy and security of your data when you use the third party software listed in this procedure. Summary As you use the tools described in this article, you notice that each one helps you troubleshoot issues at the different OSI layers. The built-in developer tools in each browser provide a quick and easy way to view HTTP content but do not let you see the details of the protocol, as do the NetLog tool and Wireshark. However, viewing qlog traces on the qvis graphical tool provides you high level trends and statistics that packet captures do not show. Using the appropriate tool with the right troubleshooting methodology maximizes the potential of HTTP/3 and QUIC for your organization.
