troubleshooting serverssl profile with client cert..
Hi all! We have a virtualserver with a serverssl profile configured with a client cert. According to the technician working with the backend nginx node they´re not getting the client cert. Does anyone know a good way for us to verify that the cert is there or not? Would a tcpdump be sufficient? /Kim
client and server ssl profiles
I am new to f5 asm, in our environment we have set up a website behind WAF in transparent mode, We have installed a wildcard certificate on real web server and replicated it on waf using client and server ssl profiles. However, when we attach this created custom profiles to virtual server site doesn't work. Interestingly, when we replace it with client/server-insecure-compatible ssl profiles site works properly. Furthermore, site works normally when we bypass waf. What steps should we take to address this issue?
Understanding the Authenticate Name Option on Server SSL profile on BIG-IP
Quick Intro Have you ever wondered what this little option on Server SSL profile really does in practice? This is what this article is all about. If you're only interested about what I learnt during my lab tests, please feel free to read just Lab Test Results section. Otherwise, enjoy the lab walkthrough :) Lab Test Results In case you just want to know what it is, this setting looks at both commonName and subjectAltName extension and if whatever name set on Authenticate Name doesn't match the name in any of these 2 fields I mentioned, certificate authentication fails and BIG-IP resets connection. Note: for Authenticate Name to work, Server Certificate has to be set to Require, i.e. BIG-IP should be configured to check the validity of Server Side certificate. On top of that, ca-file (Trusted Certificate Authorities in the GUI) is also required to be set, otherwise BIG-IP has no trusted Root CA list to validate server's certificate. What it is This is like another layer of authentication. When server sends us a Certificate and peer-cert-mode is set to require, BIG-IP looks up Root CA present in ca-file and confirms that server's certificate is trusted. Then, when authenticate-name is also set, BIG-IP checks subjectAltName extension and commonName for a match of what we've typed in this field. If no match is found, certificate authentication fails and we do not trust certificate. Otherwise, certificate is trusted and we proceed with handshake. Lab Test When Authenticate Name does not match Certificate's commonName or subjectAltName I created an end-entity X.509v3 TLS Certificate and set commonName (CN) to server1.rodrigo and subjectAltName extension to server01.rodrigo as seen below: I set Server Certificate (peer-cert-mode in tmsh) to require, added the Root CA that signed back-end server's certificate to BIG-IP's Trusted Certificate Authorities (ca-file in tmsh) and authentication-name to fail.rodrigo: On Wireshark, we see that authentication fails as soon as we receive Certificate message from server: Note: If we want BIG-IP to display the specific alert such as unknown_ca above, we need to disable generic-alert on Server SSL settings. It fails because fail.rodrigo is neither in CN nor in subjectAltName. I had set server1.rodrigo instead, remember? Let's break it down into more details: back-end server sends Certificate message to BIG-IP because peer-cert-mode is set to require, BIG-IP looks up the Root CA list in ca-file (root-ca.crt here) BIG-IP answers the following question: was back-end certificate signed by any of the certificates listed in root-ca.crt? If not, authentication fails immediately and we never get to use authenticate-name In this case it was, so BIG-IP moves on to check if fail.rodrigo is in either commonName or subjectAltName fields of back-end's X.509v3 certificate Because fail.rodrigo doesn't match server1.rodrigo, authentication fails and BIG-IP resets connection. When Authenticate Name matches Certificate's commonName or subjectAltName I've now set authenticate-name to server1.rodrigo and TLS handshake suceeds: Let's break it down into more details: back-end server sends Certificate message to BIG-IP because peer-cert-mode is set to require, BIG-IP looks up the Root CA list in ca-file (root-ca.crt here) BIG-IP answers the following question: was back-end certificate signed by any of the certificates listed in root-ca.crt? If not, authentication fails immediately and we never get to use authenticate-name In this case it was, so BIG-IP moves on to check if server1.rodrigo is in either commonName or subjectAltName fields of back-end's X.509v3 certificate Because server1.rodrigo matches server1.rodrigo in both commonName and SubjectAltName fields, authentication succeeds and TLS handshake proceeds If you ever wondered where to find commonName and SubjectAltName on TLS headers, here's where they are: The above snippet is from back-end's Certificate message that is sent to BIG-IP as part of TLS handshake. Hope that's helpful.
TLS Client Authentication from Server SSL Profile
Hi all We have a requirement to enable an outbound (internet) flow from some internal servers. Sitting near the edge of the network is an LTM that will proxy the connection from the servers, and is required to then do TLS mutual authentication (client authentication) to the target server on the internet. In this setup the LTM is, from the internal server's point of view, the server, so we configure a Client SSL Profile. All good. Next the LTM is, from the target server's point of view, a client so we configure a Server SSL Profile. Unfortunately this is not working for us. In the Server SSL profile we have set the Certificate and Key, which is the identity cert of the LTM itself signed by a 3rd party CA using a Web Server template with Client Authentication Key Usage. The logs from the target server (Apache 2.4.7) show the following: [ssl:info] [pid 5260:tid 2999946048] [client 10.128.2.109:58181] AH02008: SSL library error 1 in handshake (server server.com:443) [ssl:info] [pid 5260:tid 2999946048] SSL Library Error: error:140880AE:SSL routines:SSL3_GET_CERT_VERIFY:missing verify message My limited understanding on TLS MA is that the client should send a Certificate Verify message that proves it owns the private key. It appears the LTM is not sending this message which could explain why it is failing. I've tested a similar setup in my lab but bypassed the LTM and sure enough a Windows client does indeed send the Certificate Verify message and the transaction is successful. Any ideas on this one? Thank you.Server SSL profile Server Authentication settings don't work?
im trying to configure the server ssl profile to accept certificates which it normally wouldn't to be able to provide better feedback to the users why the connection would fail. im aware of the risk. for this i set the Server Certificate on require and the Expire Certificate Response Control and Untrusted Certificate Response Control both on ignore. still the connection fails with these messages: Jun 22 13:09:04 bigip-01 debug tmm1[17068]: 01260006:7: Peer cert verify error: unable to verify the first certificate (depth 0; cert /edit) Jun 22 13:09:04 bigip-01 debug tmm1[17068]: 01260009:7: Connection error: ssl_shim_vfycerterr:4084: unable to verify the first certificate (48) i can also set Server Certificate to ignore and then all server certificates are accepted, only then i can't use [SSL::verify_result] to determine the status, it is always 0 (OK). anyone tried this and got some more insights on how to make it work.
Change in behavior ID411879
There are quite a few threads open here in the forum, all indicating SSL handshake issues when upgrading from a version < 11.4.0 to a version >= 11.4.0. The reason for all these issues seems to be the change in behavior ID411879 (from version 11.4.0), which is described as follows: For serverssl profiles, the system uses TLS in the following way: TLS1.2, then TLS1.1 and TLS1.0. Previously, it was TLS1, TLS1.2 and TLS1.1. This might result in unexpected status settings for existing virtual servers configured in previous releases. But is there anybody, who can explain the internal logic/algorithm behind that? I mentioned it already in another thread **_what kind of sense makes an order if the second or third option will never be used, when the first one fails?_** I don't think this is a bug, but is more related to the standard SSL handshake process. But I haven't the required knowledge in detail to explain and/or understand this behavior. I hope we can finally solve this topic. Thank you! Ciao Stefan 🙂Decrypting TLS traffic on BIG-IP
1 Introduction As soon as I joined F5 Support, over 5 years ago, one of the first things I had to learn quickly was to decrypt TLS traffic because most of our customers useL7 applications protected by TLS layer. In this article, I will show 4 ways to decrypt traffic on BIG-IP, including the new one just release in v15.x that is ideal for TLS1.3 where TLS handshake is also encrypted. If that's what you want to know just skip to tcpdump --f5 ssl option section as this new approach is just a parameter added to tcpdump. As this article is very hands-on, I will show my lab topology for the tests I performed and then every possible way I used to decrypt customer's traffic working for Engineering Services at F5. 2 Lab Topology This is the lab topology I used for the lab test where all tests were performed: Also, for every capture I issued the following curl command: Update: the virtual server's IP address is actually 10.199.3.145/32 2 The 4 ways to decrypt BIG-IP's traffic RSA private key decryption There are 3 constraints here: Full TLS handshake has to be captured Check Appendix 2 to learn how to to disable BIG-IP's cache RSA key exchange has to be used, i.e. no (EC)DHE Check Appendix 1 to understand how to check what's key exchange method used in your TLS connection Check Appendix 2 to understand how to prioritise RSA as key exchange method Private key has to be copied to Wireshark machine (ssldump command solves this problem) Roughly, to accomplish that we can set Cache Size to 0 on SSL profile and remove (EC)DHE from Cipher Suites (see Appendix 1 for details) I first took a packet capture using :p modifier to capture only the client and server flows specific to my Client's IP address (10.199.3.1): Note: The 0.0 interface will capture any forwarding plane traffic (tmm) and nnn is the highest noise to capture as much flow information as possible to be displayed on the F5 dissector header. For more details about tcpdump syntax, please have a look at K13637: Capturing internal TMM information with tcpdump and K411: Overview of packet tracing with the tcpdump utility. Also, we need to make sure we capture the full TLS handshake. It's perfectly fine to capture resumed TLS sessions as long as full TLS handshake has been previously captured. Initially, our capture is unencrypted as seen below: On Mac, I clicked on Wireshark → Preferences: Then Protocols → TLS → RSA keys list where we see a window where we can reference BIG-IP's (or server if we want to decrypt server SSL side) private key: Once we get there, we need to add any IP address of the flow we want Wireshark to decrypt, the corresponding port and the private key file (default.crt for Client SSL profile in this particular lab test): Note: For Client SSL profile, this would be the private key on Certificate Chain field corresponding to the end entity Certificate being served to client machines through the Virtual Server. For Server SSL profile, the private key is located on the back-end server and the key would be the one corresponding to the end entity Certificate sent in the TLS Certificate message sent from back-end server to BIG-IP during TLS handshake. Once we click OK, we can see the HTTP decrypted traffic (in green): In production environment, we would normally avoid copying private keys to different machines so another option is use ssldump command directly on the server we're trying to capture. Again, if we're capturing Client SSL traffic, ssldump is already installed on BIG-IP. We would follow the same steps as before but instead of copying private key to Wireshark machine, we would simply issue this command on the BIG-IP (or back-end server if it's Server SSL traffic): Syntax: ssldump -r <capture.pcap> -k <private key.key> -M <type a name for your ssldump file here.pms> . For more details, please have a look at K10209: Overview of packet tracing with the ssldump utility. In ssldump-generated.pms, we should find enough information for Wireshark to decrypt the capture: Syntax: ssldump -r <capture.pcap> -k <private key.key> -M <type a name for your ssldump file here.pms> . For more details, please have a look at K10209: Overview of packet tracing with the ssldump utility. In ssldump-generated.pms, we should find enough information for Wireshark to decrypt the capture: After I clicked OK, we indeed see the decrypted http traffic back again: We didn't have to copy BIG-IP's private key to Wireshark machine here. iRules The only constraint here is that we should apply the iRule to the virtual server in question. Sometimes that's not desirable, especially when we're troubleshooting an issue where we want the configuration to be unchanged. Note: there is a bug that affects versions 11.6.x and 12.x that was fixed on 13.x. It records the wrong TLS Session ID to LTM logs. The workaround would be to manually copy the Session ID from tcpdump capture or to use RSA decryption as in previous example. You can also combine both SSL::clientrandom and SSL::sessionid which is the ideal: Reference: K12783074: Decrypting SSL traffic using the SSL::sessionsecret iRules command (12.x and later) Again, I took a capture using tcpdump command: After applying above iRule to our HTTPS virtual server and taking tcpdump capture, I see this on /var/log/ltm: To copy this to a *.pms file we can use on Wireshark we can use sed command (reference: K12783074): Note: If you don't want to overwrite completely the PMS file make sure you use >> instead of >. The end result would be something like this: As both resumed and full TLS sessions have client random value, I only had to copy CLIENT_RANDOM + Master secret to our PMS file because all Wireshark needs is a session reference to apply master secret. To decrypt file on Wireshark just go to Wireshark → Preferences → Protocols → TLS → Pre-Master Key logfile name like we did in ssldump section and add file we just created: As seen on LTM logs, CLIENTSSL_HANDSHAKE event captured master secret from our client-side connection and SERVERSSL_HANDSHAKE from server side. In this case, we should have both client and server sides decrypted, even though we never had access to back-end server: Notice added an http filter to show you both client and and server traffic were decrypted this time. tcpdump --f5 ssl option This was introduced in 15.x and we don't need to change virtual server configuration by adding iRules. The only thing we need to do is to enable tcpdump.sslprovider db variable which is disabled by default: After that, when we take tcpdump capture, we just need to add --f5 ssl to the command like this: Notice that we've got a warning message because Master Secret will be copied to tcpdump capture itself, so we need to be careful with who we share such capture with. I had to update my Wireshark to v3.2.+ and clicked on Analyze → Enabled Protocols: And enable F5 TLS dissector: Once we open the capture, we can find all the information you need to create our PMS file embedded in the capture: Very cool, isn't it? We can then copy the Master Secret and Client Random values by right clicking like this: And then paste it to a blank PMS file. I first pasted the Client Random value followed by Master Secret value like this: Note: I manually typed CLIENT_RANDOM and then pasted both values for both client and server sides directly from tcpdump capture. The last step was to go to Wireshark → Preferences → Protocols → TLS and add it to Pre-master-Secret log filename and click OK: Fair enough! Capture decrypted on both client and server sides: I used http filter to display only decrypted HTTP packets just like in iRule section. Appendix 1 How do we know which Key Exchange method is being used? RSA private key can only decrypt traffic on Wireshark if RSA is the key exchange method negotiated during TLS handshake. Client side will tell the Server side which ciphers it support and server side will reply with the chosen cipher on Server Hello message. With that in mind, on Wireshark, we'd click on Server Hello header under Cipher Suite: Key Exchange and Authentication both come before the WITH keyword. In above example, because there's only RSA we can say that RSA is used for both Key Exchange and Authentication. In the following example, ECDHE is used for key exchange and RSA for authentication: Appendix 2 Disabling Session Resumption and Prioritising RSA key exchange We can set Cache Size to 0 to disable TLS session resumption and change the Cipher Suites to anything that makes BIG-IP pick RSA for Key Exchange:Restricting number of concurrent TLS handshakes using Max Active Handshakes on BIG-IP
Introduction This is the maximum number of handshakes that BIG-IP can process concurrently per TMM. Max Active Handshakes can be used if there's an application specific reason to limit the number of concurrent TLS handshakes. It's typically used for performance or security reasons, i.e. when we know our application would only handle a specific number of concurrent TLS handshakes in a given time. BIG-IP allows us to restrict the number of concurrent TLS handshakes per TMM using Max Active Handshakes option on both Client SSL and Server SSL profiles: Do not confuse this setting with number of concurrent TLS connections. It is possible to have a much higher number of concurrent TLS connections than the number specified in Max Active Handshakes. We're literally talking about ongoing concurrent TLS handshakes where TLS Finished message has not been exchanged yet. Client SSL I've set max-active-handshakes to 1, which means that BIG-IP can only process up to 1 TLS handshake at a time. From my client machine (10.199.3.135), I used the following command to test this feature and trigger 3 concurrent TLS handshakes: Note: (10.199.3.101 is the Virtual Server) Notice that there are 3 connections here and we can distinguish them by their source TCP port. First one (37062) goes through and BIG-IP (10.199.3.101) responds correctly to Client Hello (frame #11) with Server Hello (frame #14): However, the 2 subsequent Client Hello messages sent via connection 2 (37063) and connection 3 (37066) are terminated with a TSL Fatal Alert as the maximum number of allowed handshakes configured had been reached. Also notice that the Info column on Wireshark shows that all 3 connections were received by the same tmm (tmm0). Note: TMM is BIG-IP's forwarding plane daemon and may have several instances depending on how many CPU cores exist in a given box. In a subsequent test, I sent out 2 requests via 2 different interfaces concurrently to make sure another tmm was picked: Handshake completes successfully now even though max-active-handshakes is still set to 1, because each tmm (tmm0 and tmm1) processed one a handshake which confirms max-active-handshakes counter is per tmm: Server SSL I have also set max-active-handshakes to 1 here and issued the following command from my client machine: Note: I have changed Virtual Server to 10.199.3.145 from 10.199.3.101. Here I am filtering server-side flow only to show that only the first flow where 1st handshake was already ongoing was allowed to proceed and the 2nd handshake was not allowed: If we also include the Client-side, we see that the reset (on client-side) was because of server-side reset and reset cause is Policy action: I have also confirmed that when max-active-handshakes set to 1, if a 2nd concurrent server-side handshake is triggered via a different TMM, then it goes through but I did not add the test here for brevity as it's the same test I performed on client-side. However, this also confirms that just like client-side, server-side setting works per TMM.Understanding Server SSL Trusted Certificate Authorities on BIG-IP
1 Quick Intro This article is about how this Server SSL option works on BIG-IP: We'll create a certificate chain for 2 back-end servers using OpenSSL and copy the root CA of each server to Trusted Certificate Authorities on Server SSL profile: This way, when BIG-IP is configured to validate back-end server certificates, i.e. when Server Certificate (peer-cert-mode in tmsh) is set to require, BIG-IP can trust back-end certificates that were signed by certificates in the file we added to Trusted Certificate Authorities (ca-file in tmsh). Also, because Trusted Certificate Authorities file only contain the Root CAs (and not the intermediate ones), back-end servers should also send the intermediate certificates so that the whole chain can be verified by BIG-IP. Note: for production environment, you'll probably going to use the same chain on both servers. The reason I create 2 separate chains here was just to add 2 concatenated CA's public keys to Trusted Certificate Authorities for demonstration purposes. For the purpose of this article, here's what we're going to be doing if you follow along: Creating Certificate Chain using OpenSSL Adding Certificate Chain to Apache and NGINX Adding Root CAs to Trusted Certificate Authorities on BIG-IP Confirming it all works with curl command 2 Creating Certificate Chain using OpenSSL First we create the Root Certificate Authority that is going to sign our intermediate certificate. Note that the same steps are done for server2 bundle so I'm not repeating it here. I first created a directory to keep our certificate/key files in: The next steps would be to create the CAs and the end entity certificate along with corresponding keys. 2.1 Creating server1_rootCA.key The private key (server1_rootCA.key) is the first one to be created since public key is generated from private key: Note: Are you wondering why I added -aes256 which is symmetric key whilst creating a private key, which is asymmetric key? That's only to protect the private key's contents in disk. For example, if an attacker gets hold of server1_rootCA.key, they can create a public key with such a key or decrypt any TLS traffic protected by server1_rootCA.crt. This is because our keys here are PEM encoded by default, but not encrypted. To avoid this, we can encrypt it at disk (known as encryption at rest) using symmetric key encryption with AES 256. By the way, openssl command to verify contents of private key is openssl rsa -text -in <key>.key 2.2 Creating server1_rootCA.crt Now we create our public key: Note: The required fields were filled for illustration purposes. 2.3 Creating server1_intermediateCA.key As always, we also create the private key first: 2.4 Creating server1_intermediateCA.csr Wow, wait! Why .csr? Why not .crt like we did before? The reason is because our intermediate CA will be signed by our Root CA, remember? We'd only create a .crt directly if we were creating a self-signed certificate like our Root CA. The idea of the .csr is that we'll set up all the information we want our certificate to have and we'll use our Root CA to turn it into a .crt file. Does it make sense? 2.5 Creating server1_intermediateCA.crt Based on the information from server1_intermediateCA.csr, we can now use server1_rootCA.key to create server1_intermediateCA.crt. However, before we do that, we may want to add some capabilities to our intermediate certificate which is recommended by man x509v3_config command: Note: the flags above are out of the scope of this article but setting basicConstraints to critical means that for validation purposes, the flags we're setting have to be taken into consideration. For example, CA:true means that this is a CA capable of signing other certificates such as our end-of-chain certificate that we'll create after that. We'd set this flag to false when creating end of chain client or server certificates as they're not supposed to sign other certificates. The keyUsage indicates what this CA should be used for. We're now using these extensions to create and sign server1_intermediateCA.crt using Root CA's private key: This way, if BIG-IP has Root CA's public key, it can confirm that this certificate was indeed signed by Root CA's private key. 2.6 Creating server1_endofchain.key Just like Root and Intermediate CAs, we create private key first: 2.6 Creating server1_endofchain.csr We now type the info we want our .crt file to have once it's signed by our intermediate CA: 2.6 Creating server1_endofchain.crt We also have an extension file here with the typical back-end server flags and constraints that can be found in man page as well: And we now create server1_intermediateCA.crt: We can also verify csr/crt/key as they should all output the same md5 hash: Note: For server2, we use the same steps but replacing server1 with server2 when appropriate. 3 Adding Certificate Chain to WebServer 3.1 Apache If not enabled, activate ssl module using a2enmod command: Don't restart Apache just now because we'll first add our end entity certificate and intermediate certificate to Apache's config file first. First I'll the default ssl file from sites-available to sites-enabled because only files in sites-enabled are actually active on the webserver. As Apache already has a default template, I'll create a symbolic link as this is just for demonstration purposes: And the only thing I did was to edit SSLCertificateFile, SSLCertificateKeyFile and SSLCertificateChainFile and add our end entity certificate, end entity key and intermediate CA certificate respectively: Once we try to restart Apache, we need to type in our passphrase for the certificates encrypted with AES256 at rest, remember? Otherwise, Apache won't be able to read it. We now confirm Apache is running properly: And most importantly, that we can reach our SSL protected website locally: And from BIG-IP: Note: the k flag on curl command skips certificate validation and for the moment that's what we want. Even if we add certificate validation to BIG-IP configuration, we need to make sure we understand that our curl command above is executed from BIG-IP's control plane (Linux) so a certificate added to BIG-IP's configuration file only applies to BIG-IP's forwarding plane (tmm). Therefore, even after we add the correct valid certificate to Trusted Certificate Authorities, the curl command above will not automatically reference it. 3.2 NGINX If we're using NGINX webserver, we should bundle our end entity certificate together with intermediate CA's certificate: And add it to your webserver's configuration on /etc/nginx/conf.d/ directory: In this case, NGINX will send 2 certificates (end entity and intermediate one) back to BIG-IP. FYI, a bundled chain is just one PEM certificate concatenated with the other: Not that hard to picture, is it? And lastly, we reload NGINX: For more details, please have a look at NGINX documentation. 4 Adding Root CAs to Trusted Certificate Authorities on BIG-IP 4.1 What do we add to Trusted Certificate Authorities? One important concept to grasp is that we don't need to copy the whole certificate chain to BIG-IP. The chain belongs to your back-end server. The only thing we need from the chain to add to BIG-IP is the top-level Root certificate(s) that signed the chain BIG-IP will verify. From the point of view of BIG-IP, we'll make it trust whichever certificate is signed by CA list added to Trusted Certificate Authorities. Therefore, we only need to add server1_rootCA.crt and server2_rootCA.crt in our lab test. I have copied both Root CAs from Server1 and Server2 to a temporary directory on BIG-IP: I'll make it a bundle named servers_rootCA.crt: 4.2 Importing Root CA to BIG-IP 4.2.1 Via GUI We can now import it to BIG-IP. In the GUI on v15.x, you'd go to Certificate Management → Traffic Certificate Management → SSL Certificate List → Import: Select Import Type → Certificate: And upload (or copy/paste it if you wish) the contents of servers_rootCA.crt: 4.2.2 Via TMSH As BIG-IP versions change, GUI is more susceptible to changes. If that is the case and this article ever gets outdated, here's the tmsh command equivalent to import certificate chain: 4.3 Adding Chain to Trusted Certificate Authorities If we go to our Server SSL profile we should now see servers_rootCA.crt as option to add to Trusted Certificate Authorities: And we just click on Update. Note: notice that Server Certificate is set to require! Don't forget that! 4.4 Final test with Curl command For this test I disabled Generic Alert on Server SSL profile: Otherwise, BIG-IP would not tell us the real reason why handshake failed. 4.4.1 Trusted Certificate Authorities set to None When I issued curl command from my client it failed as expected because Server Certificate is set to require and we added nothing to Trusted Certificate Authorities: And I took the opportunity to take a tcpdump capture in parallel: And on Wireshark we can clearly see the reason why our curl command failed: BIG-IP doesn't recognise the CA as trusted because it's not on Trusted Certificate Authorities. In fact, there's nothing there so any back-end certificate would fail anyway. 4.4.2 Trusted Certificate Authorities set to servers_rootCA.crt Now our curl command works: I also took a tcpdump capture here: In fact, one thing that is interesting to note is that we can see that Certificate message sent from back-end server only has the intermediate CA and the end entity certificate: If we wanted, we could've configured Apache or NGINX to add Root CA to the bundle but it's not necessary. The reason why is because BIG-IP already has the Root CA in the bundle we added to Trusted Certificate Authorities, remember? That's it for now. Appendix 1 When your certificate is signed by a well-known Root CA Web browsers have somewhere close to a thousand trusted Root CAs by default. These were added by the developers of your web browser. BIG-IP has a default file called ca-bundle.crt with the usual browser's trusted CA repository and on v15.x it contains 857 trusted CAs: If your certificate was signed by one of the major CAs out there, it's likely the only thing you'll need to do in production would be to use ca-bundle.crt: Why? Because if the Root CA that signed your certificate is already in ca-bundle.crt then there's no need to use a different bundle. It's up to you, really!
