Support dynamic CRL check for clientSSL profile (BIG-IP 15.1)
Hi, Did anyone tested (dynamic) CRL validator object for client SSL profile? (BIG-IP v15.1): It should work in v 15.1 (fixed bug 743758 - https://cdn.f5.com/product/bugtracker/ID743758.html ) I'm getting following errors for all client certificates: err tmm1[21207]: 01a40008:3: Unable to build certificate trust chain for profile /clientssl_profile tmm1[21207]: 01260009:4: clientIP:62042 -> VIP:443: Connection error: ssl_hs_do_crl_validation:6014: alert(46) unknown certificate error With CRL File it works ok, but file does not automatically fetch, check, and cache CRL files… Kr, EPX
Sample Linux script to update CRL file from Certificate Authority
Problem this snippet solves: CRL files are signed lists of revoked serial numbers issued by a specific Certificate Authority (Verisign, Godaddy, GlobalSign, etc). There are several advanced methods of dealing with revoked certificates, the best of which is OCSP stapling. Other methods are OCSP responders or CRL Distribution Points. However, for small or internal projects, some administrators rely on simply using straight-up CRL files. After a time, the administrator will realize he or she needs to automate this process with a script. The administrator can automate this process on the BIG-IP itself with an iCall script. Here's a link to a great example of the iCall solution. However, some administrators many need to use a straight-up Linux device to pull and copy the CRL files around to many different devices, only one of which is the BIG-IP. How to use this snippet: This is a sample of a Linux script that pulls down a CRL file from GoDaddy, verifies it and then copies it to BIG-IP. Ensure that the Linux device can copy files directly to the BIG-IP via ssh-key authentication. Modify the 'f5' variable of the script to point to the BIG-IP. If not using GoDaddy, find the URL of the CRL file for the appropriate CA. If the 'ssl-crl' object hasn't been created on the BIG-IP yet, then it must be done manually the first time. Download the CRL, copy it to the BIG-IP's /var/tmp area. Then login to the BIG-IP and issue the following command: tmsh modify sys file ssl-crl gdcrl source-path file:/var/tmp/CRL After that, the script should work. Code : #!/bin/bash # # script to download a CRL from GoDaddy CA. # See this page for GoDaddy CRL information: # https://certs.godaddy.com/repository # Verify CRL # Convert CRL # Copy to BIG-IP # exit on error to prevent copying corrupt CRL to BIG-IP set -e f5=yourbigip.com f5port=22 crlurl=https://certs.godaddy.com/repository/mastergodaddy2issuing.crl gdcrturl=https://certs.godaddy.com/repository/gdig2.crt gdcrt=gdig2.crt echo "Automated CRL update Script" echo "Downloading from $crlurl" echo "Copying to ${f5}:${f5port}" echo "Last line should be SUCCESS" echo "---- GO ----" if [ ! -f $gdcrt ]; then echo "Fetching GoDaddy Certificate" curl $gdcrturl > $gdcrt fi cf=gdroot.crl last=last.crl if [ -f $cf ]; then if [ ! -s $cf ]; then echo "Found existing zero-length CRL file, deleting" rm -f $cf else echo "Found existing $cf - moving to backup" mv $cf $last fi fi echo "Downloading CRL file $cf" curl $crlurl > $cf echo "Testing $cf for readability" test -f $cf test -r $cf echo "Testing to see if $cf is zero length" test -s $cf if [ -f $last ]; then echo "Testing if $cf is newer than backup file $last" if [ ! $cf -nt $last ]; then echo "File not changed. SUCCESS" fi fi echo "Verifying CRL against certificate, converting to PEM format" openssl crl -inform DER -CAfile $gdcrt -in $cf -outform PEM -out ${cf}.pem echo "Testing PEM for zero length" test -s ${cf}.pem echo "Copying CRL file to bigip" scp -P ${f5port} ${cf}.pem root@${f5}:/var/tmp echo "Importing CRL into system" echo " this may fail if object was never created - replace 'modify' with 'create'" ssh root@${f5} -p ${f5port} "tmsh modify sys file ssl-crl gdcrl source-path file:/var/tmp/${cf}.pem" echo "SUCCESS"Security Sidebar: My Browser Has No Idea Your Certificate Was Just Revoked
Encryption is a fundamental reality on the Internet today. Most sites use SSL/TLS for encryption, and you can identify these sites by the https:// in the address bar of your browser. The Internet security service company Netcraft has been tracking SSL usage for over 20 years now, and their most recent data shows that there are now more than one thousand times more certificates on the web today than in 1996. DevCentral is no exception to this SSL phenomenon…go ahead, check your browser’s address bar and notice the address for this article (or anything else on DevCentral for that matter) will start with https:// instead of plain old http://. This SSL/TLS encryption provides a secure means of communication between your browser and the web server. In order to make all this encryption happen, encryption keys are shared between the web server and your browser. Encryption key exchange gets very complicated and this article is not meant to explain all the details of encryption key exchange mechanisms, but from a very high-level perspective, it’s fair to say that these keys are shared by using the web server’s SSL/TLS certificate. When a user visits a secure website, an encryption key exchange process takes place, and the resulting encryption keys are used to encrypt all communication between that user and the web server. A certificate is a digital file that holds several pieces of information related to a particular website. One of the pieces of information it holds is the public portion of the encryption key used to encrypt all the communications to/from the web server. Another piece of information it holds is the effective dates of the certificate. After all, these things are only good for a finite period of time (typically 1-2 years). In a perfect world, a web server would be issued a certificate and that certificate would never get compromised and it would be used for the full duration of the life of the certificate. But we don’t live in a perfect world. The reality is that certificates get compromised all the time, and when that happens, the certificate needs to be revoked. Typically when a web server certificate is revoked, a new certificate is created and used in place of the old, revoked certificate. But, how does a user know that a certificate has been revoked? The Magic of CRL and OCSP Here’s how it works…when a user visits a secure website, the certificate is sent from the website to the user’s browser (Chrome, Firefox, Internet Explorer, Safari, etc). Because certificate sharing creates significant computational overhead, many browsers simply store the certificate information from a previously-visited website in their cache so they don’t have to keep asking for a new certificate each time they visit that website. This is nice because it significantly speeds up the user experience for loading that particular secure website, but it also presents a problem when the certificate is no longer valid. In order to check that a given certificate is still valid, the concept of a Certificate Revocation List (CRL) was introduced. The CRL is a digital file created by a Certification Authority (the organization that creates and distributes certificates) that contains the serial number for each certificate that has been revoked by that CA. In order for a browser to check that a given certificate is still valid, the CRL must be downloaded and the serial number for the website you are visiting must be checked against the CRL to ensure the certificate is not revoked. If the certificate is not revoked, all is good and the browser displays the page. But if the certificate has been revoked, the browser should display a warning page that tells you the certificate has been revoked. Some browsers will allow you to continue to the page anyway and others won’t…it just depends on the browser. The CRL check method is computationally expensive because the browser has to download the CRL every time it needs to check a certificate (which happens very frequently), and it also has to search through the CRL for a match of the serial number of the certificate it is using (CRL files can get extremely large). In order to avoid frequent CRL downloads and searches, some browsers will cache the CRL for a given period of time and simply check the cached CRL instead of downloading a new one each time. This helps speed things up, but what if the CRL changed since the last time it was cached in your browser? This situation became a big enough problem that a new, faster solution was introduced. The Online Certificate Status Protocol (OCSP) was developed in 1999 and it is a solution that queries an online database of serial numbers for revoked certificates. Rather than host CRL files, CAs can instead setup an OCSP server and include its URL in issued certificates. Clients that support OCSP can then query the database to see if a given certificate has been revoked, instead of downloading the entire CRL file. OCSP is a much more efficient solution than the CRL method. Here’s a quick view of the certificate issued to f5.com. Notice it was issued by the Entrust Certification Authority, therefore Entrust is the one who manages the CRL file for all their revoked certificates and they also manage the list of serial numbers used in the OCSP queries. The picture on the left is the detailed view of the f5.com certificate with the CRL location listed. This is the URL where the browser can go download the CRL from Entrust. Keep in mind that this CRL is just the one managed by Entrust. I took the liberty of visiting the URL where the CRL is located, and the details for the CRL are shown in the picture on the right. Notice that the CRL is simply a big list of serial numbers with revocation dates and reasons for revocation. The following screenshot is the OCSP portion of the f5.com certificate. Notice the URL listed in the details section at the bottom of the screenshot. This is the URL that the browser will visit in order to check the current certificate serial number against the database of revoked serial numbers managed by Entrust. So Many Certificates, So Little Time… Now that we understand how a browser checks to see if a certificate is still valid, let’s take a little deeper look at the different types of certificates available today. There are three types of certificates you can purchase today: Domain Validation (DV), Organization Validation (OV), and Extended Validation (EV). The DV certificate is the cheapest and most popular type of certificate. Each CA makes up its own rules as to what they require from an organization before they will issue the DV certificate. Typically it’s a very simple (and many times, automated) process like the CA sending you a file and you placing that file on the webserver at the domain in question…just something simple to let the CA know that they are issuing a certificate for a given domain to the actual owner. There are, of course, many different ways to hack this process and get a DV certificate for a domain that you don’t own. But, that’s a topic for another day. The OV certificate is more expensive than the DV, and it takes things a bit further with respect to the CA checking that the requesting organization actually owns the domain name. There are some other organizational vetting procedures that a CA might take in order to more fully understand that the requestor is the owner. In addition to what they would do for a DV certificate, maybe they’ll make a few reference phone calls and do some simple background checks. But, then again, it’s totally up to the individual CA to develop their own procedures for vetting an organization prior to issuing an OV certificate. The EV certificate is the most expensive, and it requires the most amount of background checking and validation from a CA before it is approved. In fact, the process for vetting an organization prior to issuing an EV certificate is governed by the CA/Browser Forum. This organization has developed a robust list of requirements that must be met before a CA can issue an EV certificate. Companies that want the EV certificate do so because they want to show the world that they are serious about the security of their online presence. The reality is that users visit these secure sites via an Internet Browser (Chrome, Firefox, Internet Explorer, etc), so it’s interesting to see how these different browsers handle certificate revocation and also how they treat the different kinds of certificates. The top 3 browsers on the Internet today are Google Chrome (70.4%), Mozilla Firefox (17.5%), and Microsoft Internet Explorer (5.8%). Almost 94% of all Internet traffic is displayed by one of these three browsers. Let’s take a quick minute to see how these check for certificate revocation. Google Chrome Google Chrome is by far the most popular browser today. When it comes to certificate revocation checking, Chrome blazes its own trail and does its own thing. The CRL and OCSP methods of certificate revocation checking are the industry standards, but Google isn’t standard in this space. It has created what’s known as a CRLSet to check for certificate status. A CRLSet is Google’s own list of revoked certificates that it compiles and updates when it crawls the CRLs from the major CAs around the world. Instead of checking an OCSP responder or a CRL, Google Chrome simply checks its own CRLSet for certificate status when visiting a secure website. Google claims that this is faster and safer for the user than the traditional CRL or OCSP methods. Some people think this approach is good because it’s faster to check a locally stored list than using the traditional methods and you also don’t have to worry about OCSP responder or CRL distribution point availability. But others are skeptical because the CRLSet is only comprised of certificates that Google deems worthy to include. What if the certificate you need to check isn’t on the CRLSet list? Also, the CRLSet file size is explicitly limited to 250KB. If something happens and lots of certificates are suddenly revoked causing the CRLSet to get bigger than 250KB (Heartbleed for example), then certificates are deleted from the CRLSet so that it stays at the 250KB max size. What if the certificate status you need gets deleted from the CRLSet during one of these bloat sessions? Mozilla Firefox Firefox allows you to check for revoked certificates via the OCSP method, but it doesn’t use the CRL at all. If a given certificate includes an OCSP address in the Authority Information Access (AIA) portion of the certificate, then Firefox will query the OCSP server to make sure the certificate is not revoked. If the OCSP server isn’t available or if the OCSP address is not present in the AIA field of the certificate, then Firefox won’t check revocation status and will present an error message (which the user can click through to proceed anyway). Check out the screenshot below for the settings in Firefox (version 46.0.1). Microsoft Internet Explorer Interestingly, Microsoft lE conducts the most comprehensive certificate revocation check of these leading browsers. The default setting is like Firefox…it checks the OCSP responder if the address is present in the AIA field of the certificate. But, if the OCSP server is not available or if the OCSP address is not present, it will then check the CRL (it checks the CRL loaded in cache if possible so that it doesn’t have to continually download a large CRL file). If neither of these is available, it will present a warning page and give the user an option of either proceeding forward with an unknown certificate status or closing out of the browser. The following screenshot shows the settings for certificate revocation checks in IE: You can see that each browser handles certificate revocation a little different than the next. So, it’s entirely possible that a revoked certificate could fall through the cracks if Google decided not to add it to their CRLSet, Firefox couldn’t contact the OCSP server, and Internet Explorer had an outdated version of the CRL stored in cache. Be careful out there...
SSL client profile - certificate authentication - multiple CRL files
Hi guys, currently I'm running a tests with certificate-based user authentication, using LTM/APM. In general everything is working fine, except for the fact, that there is no option to check several CRL files in one SSL client profile. As there are multiple CAs, that have issued client certificates, I need to check several CRL files. The documentation is not very specific about this piece of information. There are only statements, that it is not allowed to have multiple CRLs in a single master file. I have tried to use CRLDP, but this does only work in conjunction with LDAP. I can only provide the CRLs via file upload to BIG-IP or via HTTP downloads from an internal server. The only idea I have so far, but which is still not tested, is to use several SSL client profiles, one for each Trusted CA, assign the correct CRL file, stored locally on the BIG-IP, and the assign the SSL client profiles dynamically, based on the requested hostname in the SNI extension. To be honest, I cannot believe that there is no easier way to achieve this. Any ideas on that? Thanks in advance. Greets, svs
iCall CRL update with Route Domains and Auto-Sync
Problem this snippet solves: iCall script to update CRL file within F5 BIG-IP when the HTTP request must run from a specific Route Domain and also uses logger to write logs to the default LTM location. The original was to also update an iFile of the CRL file for use within an iRule however I have removed that due to it being a very special case (I may add another snippet later to detail that one). Important point here is we update the CRL file located within a folder (or partition) that was linked to a Sync-Only Device Group with auto-sync enabled e.g. CRL files are created and saved to /Common/ crl / This way the iCall script does not need to trigger any sort sync and the rest of the configuration can be left as manual sync. Code : sys icall handler periodic /Common/someCrl-CrlUpdate { arguments { { name rd value 2 } { name url value https://172.31.0.1/somepath/to/crlUpdateFile.crl } { name host value somecrl.CADomein.com } { name folder value tempCrlDirectory } { name sslCrl value /Common/crl/someCrlFile.crl } } interval 600 script /Common/iCallCrlUpdate } sys icall script /Common/iCallCrlUpdate { app-service none definition { set logTag "iCallCrlUpdate" set logLevel "notice" # Getting handler provided arguments foreach arg { rd url host folder sslCrl ifileCrl } { set $arg $EVENT::context($arg) } # Create a directory to save files to disk set crlDir /var/tmp/$folder exec mkdir -p $crlDir exec /bin/logger -i -t $logTag -p local0.$logLevel "Running, CRL URL=$url, Host=$host, SSL CRL=$sslCrl, iFile CRL=$ifileCrl, Directory=$crlDir, rd=$rd" # Download CRL file from provided route domain (rd) and url arguments and save to temporary directory set status [exec /usr/bin/rdexec $rd /usr/bin/curl-apd -s -o $crlDir/LatestCRL.crl -w %{http_code} -H Host:$host $url] if {$status == 200} { # Update F5 SSL CRL file tmsh::modify sys file ssl-crl $sslCrl source-path file:$crlDir/LatestCRL.crl exec /bin/logger -t $logTag -p local0.$logLevel "F5 CRL files update complete." } else { exec /bin/logger -i -t $logTag -p local0.error "Command /usr/bin/rdexec $rd /usr/bin/curl-apd -s -o $crlDir/LatestCRL.crl -w '%{http_code}' -H 'Host: onsitecrl.trustwise.com' $url, failed with status=$status" } } description none events none } Tested this on version: 12.1
F5 BIG-IP LTM VE - disk space issue
We do have a cron job running which updates a CRL file on regular basis in order to allow F5 to use up to date CRL file for verification of client certificate during mTLS. What we have noticed is, that the HDD space is constantly being filled up. We did some investigation and we found out, that there are always files being created in the ./config/filestore/.trash_bin_d/ folder Please see below an example ls -lh ./config/filestore/.trash_bin_d/.backup_1597885592_196_d/Common_d/certificate_revocation_list_d/:Common:trust2408-full_88715_317 ls -lh ./config/filestore/.trash_bin_d/.backup_1597885592_196_d/Common_d/certificate_revocation_list_d/:Common:trust2408-prod_79040_294 trust2408-prod / trust2408-full are the certificate revocation lists created by the script. This is causing, that the file system below is full and F5 can no longer operate properly Here is a difference within 24 hours Status from 19/08/2020 Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg--db--vda-set.2._config 2.1G 467M 1.5G 24% /config Status from 20/08/2020 Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg--db--vda-set.2._config2.1G805M1.2G41% /config I see that there are 10 files generate in less than 24 hours, each file has 43M. At the end of the day it will eat up all free space and cause F5 to not function properly any longer. I would expect, that these files which are in "./config/filestore/.trash_bin_d/.backup_nnnn " foler are automatically removed by F5 on regular basis. Or am I missing anything here? We do have 4 F5 load balancers which have this script enabled and only 2 of them have this issue. All the setup is the same. Has anyone experienced anything similar? Any advice is highly appreciated.
APM CRL checking
Hello. I'm working with APM and using On-demand client certificate auth with policy type - request, because i want to have fallback rule in case of certificate validation failure. So, according to http://support.f5.com/kb/en-us/products/big-ip_apm/manuals/product/apm_config_10_2_0/apm_config_clientcert_auth.html - You should not configure CRL updates if you are using the Access Policy Manager to generate and issue On-Demand Certificates to users (using either a self-signed client root CA certificate, or a client root CA certificate from a trusted CA). In this case the Access Policy Manager manages CRLs internally. But seems it's not working, i have certificate with public crl in it like http://CA/revoke.crl, i can resolve this address through BIG-IP, but if i revoke some cert its still valid for authentication so i assume that crl isn't checked. So how to setup crl checking for APM if i have only URL where Crl file located and could not provide ldap crldp or OCSP?Create Your Own Certificate Authority
Problem this snippet solves: The main goal of this article is to share an easy way to create your own Certificate Authority (CA) for your lab enviroment with APM module. REF - https://github.com/DariuSGB/LabCA This repository is composed by a set of scripts that give you an easy way to: Create your own root CA. Create your own intermediate CA, signed by your root CA. Create your own certs, signed by your intermediate CA or your root CA. Create your own OCSP cert, for using it in your OCSP responder. Create your own CRL cert, for using it directly in your APM. Revoke your certs (remember to refresh your CRL cert after that). Create your own PKCS#12 cert (from regular PEM certs/keys) for installing it in your windows enviroment. Invoke a OCSP responder of your certs enviroment (remember to create a OCSP cert first). How to use this snippet: Download and install your enviroment using these commands: git clone https://github.com/DariuSGB/LabCA.git cd LabCA chmod +x $(ls | grep -v README) Tested this on version: 14.1How to understand "Update CRL" field in LTM's Certificate Authority profile?
Hi, I am wondering, how should I understand the CRL related fields in LTM's CA profile (Local Traffic > Profiles > SSL > Certificate Authority)? I cannot find any documentation for this profile. I would like to use this CA profile in APM's machine certificate check and make sure that machines with revoked certificates are denied access to APM resources. I understand that the "Certificate Revocation List (CRL)" field defines a static CRL file stored locally which I previously created under System > File Management > SSL Certificate List. But using CRL without a periodic and automatic update does not make sense. But I am struggling to understand what exactly would LTM do if I check the "Update CRL" checkbox under CA profile. How would LTM update this CRL? Where would it get the information for this update from? Would it read the CRL location URL from the certificate and try to access it? Anybody has an idea on how this is meant to work?
