Forum Discussion

_JOHN_'s avatar
_JOHN_
Icon for Altocumulus rankAltocumulus
Aug 10, 2022

CRL Validator

From v15.1 onwards client SSL profiles support CRL validator objects as per this bug report:

Bug ID 743758 (f5.com)

I have no experience of CRL Validator.  I have just started to read about it, but I haven't found much info and would appreciate input from those who already have experience to help me properly understand its operation and benefits/limiataions.

The following devcentral post discusses it:

Support dynamic CRL check for clientSSL profile (B... - DevCentral (f5.com)

The F5 client SSL overview pages don't look to cover it, but the following describes it from the server SSL profile point of view (heading "Create a custom Server SSL profile that supports CRL"):

AskF5 | Manual Chapter: SSL Traffic Management,SSL Traffic Management,SSL Traffic Management,SSL Traffic Management,SSL Traffic Management

So first off let me just make sure I have understood the basic setup (please correct anything I have wrong):

1) Client SSL profile set up as normal for client authentication e.g. require a cert, and attached to relevant VS.
2) Internal proxy created referencing a DNS resolver.  The DNS resolver is used to map any of the expected DNS zones referenced in CRLDPs to DNS servers that can perform the name resolution.
3) CRL object created referencing internal proxy.
4) CRL object attached to client SSL profile mentioned in point 1.

In this setup the F5 will examine the CRLDP contained within the client cert, perform a DNS lookup for it, and then progress to download the CRL to use for checking the client cert.  So I have the following pieces I want to understand (answers for any appreciated :-)......
> How are different methods e.g. LDAP/HTTP handled?
> What is the approach if multiple CRLDPs are specified in the cert?
> What happens if the CRLDP isn't covered by the DNS zones in the DNS resolver?
> Is there any mechanism to restrict the set of CRLDPs that this download and cert check will operate for, i.e. only perform the operation if the CRLDP in the cert matches a specific set of known CRLDPs (I assume not in the normal setup, it would need to be performed via the likes of an iRule instead?)
> The CRL object has a setting "Strict Revocation Check".  The operation of this isn't clear to me - it seems like if it is set the client connection is paused until the CRL is downloaded and checked, or if not set the F5 starts the download but returns 'unkown' for the cert revocation check - if that is the case will the SSL connection be allowed to establish anyway or how is the unkown status handled?
> What sort of error handling is used, e.g. what if the F5 can't connect to the CRLDP - is there a timeout value used?  What if it does connect but is taking a long time to download - will the SSL handshake be paused indefinitely (I assume based on the setting of Strict Revocation Check?) - if so have folks needed to increase the SSL handshake timeout setting in the SSL profile?
> The CRL is cached - is that governed by the contents of the CRL itself, e.g. nextupdate filed, or is it configurable?
> Is there any way to manually specify specific CRLDPs that will be downloaded and constantly refreshed whether certificates which reference them are received or not?

The last query is the main reason I am looking into the CRL Validator setup in the first place.  I have a use case whereby client certs can be presented which will be signed by a few different CAs, and therefore multiple CRLs need to be checked not just one.  I have seen comments that this can be achieved by downloading the different CRLs (the CRLDPs are already known) and simply concatenating them (in PEM format) - e.g. done periodically by iCall or bash script and the concatenated 'bundle' is then specified as the crl-file on the client SSL profile.  However I raised an F5 support request to check if the F5 does definitely support concatenated CRL files in the client SSL profile and unfortunately the answer was no 😞

application delivery
security
  • Kevin_Stewart's avatar
    Kevin_Stewart
    Icon for Employee rankEmployee
    Aug 22, 2022
    > How are different methods e.g. LDAP/HTTP handled?
    [Answer] I'm pretty sure this supports both.

    > What is the approach if multiple CRLDPs are specified in the cert?
    [Answer] Does that ever happen? But if so, it would attempt process the first one.

    > What happens if the CRLDP isn't covered by the DNS zones in the DNS resolver?
    [Answer] The DNS resolver is intended to provide local DNS resolution for the CRLDP URL. You can simply create a DNS resolver that uses "." as the name (no quotes) to cover all URLs.

    > Is there any mechanism to restrict the set of CRLDPs that this download and cert check will operate for, i.e. only perform the operation if the CRLDP in the cert matches a specific set of known CRLDPs (I assume not in the normal setup, it would need to be performed via the likes of an iRule instead?)
    [Answer] You should be able to point the "Internal Proxy Name" at a local VIP and process the requests with an iRule.

    > The CRL object has a setting "Strict Revocation Check". The operation of this isn't clear to me - it seems like if it is set the client connection is paused until the CRL is downloaded and checked, or if not set the F5 starts the download but returns 'unkown' for the cert revocation check - if that is the case will the SSL connection be allowed to establish anyway or how is the unkown status handled?
    [Answer] This basically boils down to timing. CRLs are binary files that have to be downloaded in full and then parsed, and CRLs can be large. It's likely in some circumstances that a CRL file could be so large that it could delays requests for an unacceptable amount of time. So this option basically says, if the file is big, trigger the download, and while it's downloading return an unknown (revocation) response. That unknown will probably cause the first or first set of responses to fail depending on how you've configured things, so ultimately you have to wait for the CRL to finish downloading anyway. The CRL is cached so that subsequent requests don't have this issue.

    > What sort of error handling is used, e.g. what if the F5 can't connect to the CRLDP - is there a timeout value used? What if it does connect but is taking a long time to download - will the SSL handshake be paused indefinitely (I assume based on the setting of Strict Revocation Check?) - if so have folks needed to increase the SSL handshake timeout setting in the SSL profile?
    [Answer] If DNS resolution fails, it returns an error response immediately. If DNS succeeds but the request fails, it'll make a small series of attempts (3 or 5 I believe), then return an error response.

    > The CRL is cached - is that governed by the contents of the CRL itself, e.g. nextupdate filed, or is it configurable?
    [Answer] I'm not entirely certain if it's configurable, but I believe the cache timeout is based on the CRL's original nextupdate value.

    > Is there any way to manually specify specific CRLDPs that will be downloaded and constantly refreshed whether certificates which reference them are received or not?
    [Answer] You could always run a script to periodically issue a request that triggers a CRL download. But honestly, that's what OCSP is for. 😉
    • _JOHN_'s avatar
      _JOHN_
      Icon for Altocumulus rankAltocumulus
      Sep 06, 2022

      Firstly – sorry for the delay in replying, I was away for a while.

      Secondly – thank you very much for the in depth reply, much appreciated 😊

      Regarding the last point:

      > Is there any way to manually specify specific CRLDPs that will be downloaded and constantly refreshed whether certificates which reference them are received or not?

      [Answer] You could always run a script to periodically issue a request that triggers a CRL download. But honestly, that's what OCSP is for. 

      I don't really get the reference to OCSP here.  In my scenario the F5 is the server end of the communication receiving a client certificate during mutual authentication, and I know all of the CAs that client certs align to (plus their CRLDPs).  OCSP would let me check on an individual cert by cert basis, but it would therefore hold up every SSL handshake while the revocation check occurs.  CRL however seems much more suited to this type of server setup and provides benefits over OCSP – client certs are compared against a local copy of the CRL file, so the time to check will be quicker than OCSP (providing you have a cached copy of the CRL).  The downside is that CRLs may be cached for a long time which could mean revocation status is missed.  However if you actively download the CRLs continually in the background (e.g. every 15 minutes) then there would be many benefits over OCSP as I see it:

      • Other than maybe on the first ‘run’, you don't hold up the client connection while an outbound call is made to a revocation checking service – so client connection time is much better.
      • Any issues with the CRLDPs, or the network connectivity to them, will not cause immediate problems for the client connection – it will continue to be checked against the latest copy of the relevant CRL, which will be updated when the CRLDP issue recovers (would likely want to be able to collate logs to pick up on any long lived problems).  However if instead using OCSP the connections would be blocked until the responder becomes available again.
      • Assuming the CAs publish new CRLs when certs are revoked then by continually downloading them on your own schedule rather than waiting for the nextupdate time you will have a very up to date mechanism for checking revocation.
        For Example:  Reading the second ‘Note’ on this AWS documentation page:
        Setting up a certificate revocation method - AWS Certificate Manager Private Certificate Authority (amazon.com)
        I would presume that if one of the CAs in question is the AWS certificate manager then using continually downloaded CRLs would actually pick up on revoked certs quicker than using OCSP.  It would obviously depend on the specific CAs in operation, but in my scenario (which I expect will be a common one) when you know the CAs then you can determine their method of operation and I find it likely that CRL may come out as being much better than OCSP providing it can be used appropriately.

      Unfortunately I don’t see a way to specify the likes of an ‘update interval’ with CRL validator, so I am assuming there isn’t a way to keep the CRL fresh and you have to just rely on the cache expiring (likely based on CRL nextupdate field) and then a new download being triggered by the next incoming client connection that references that CRL.  This wouldn't really suit in my scenario because each time the CRL expires I will potentially drop some connections while a new CRL is downloaded, plus there would be too long a period during which revoked certificates may end up being trusted - not to mention the potential problems if the CRLDPs couldn't be reached for some reason.  Hence my question about the ability to use some form of background download of the CRLs to keep them fresh and avoid the need to repeatedly hold up client connections to download a new CRL.

      Regarding the statement “You could always run a script to periodically issue a request that triggers a CRL download” – do you mean there would be a way to tie this in with CRL Validator to essentially provide the functionality I have described, i.e. CRL Validator is deployed against the VS, but then some form of script (iCall maybe?) keeps the downloaded CRLs ‘fresh’ in the background?

      Also thank you for the second reply with info on concatenated CRLs – I have tested this and found it to work, however I would need it to actually be an approved F5 approach or otherwise I can’t rely on it.

      BTW – I have definitely seen cases where more than one CRLDP is specified in a client cert.  Sometimes this provides both LDAP and HTTP DPs.  Sometimes it just provides redundant DPs using the same protocol – e.g. the current amazon.com cert (presented to UK users anyway) is signed by DigiCert and contains:

      [1]CRL Distribution Point
      Distribution Point Name:
      Full Name:
      URL=http://crl3.digicert.com/DigiCertGlobalCAG2.crl

      [2]CRL Distribution Point
      Distribution Point Name:
      Full Name:
      URL=http://crl4.digicert.com/DigiCertGlobalCAG2.crl

  • LiefZimmerman's avatar
    LiefZimmerman
    Icon for Admin rankAdmin
    Aug 15, 2022

    _JOHN_ - thanks for the context and effort you've put into this question - I will see if I can find someone who knows about this to take a look.

  • Kevin_Stewart's avatar
    Kevin_Stewart
    Icon for Employee rankEmployee
    Aug 22, 2022

    Forgot to mention, that while it isn't technically "supported", concatenated CRLs can work in some scenarios.

  • Kevin_Stewart's avatar
    Kevin_Stewart
    Icon for Employee rankEmployee
    Sep 06, 2022

    OCSP does indeed have a few advantages over CRL:

    • CRLs are often large, so they would have to be downloaded in entirety before parsing. That could cause some initial connections to fail while that's happening. Or you have to go out of your way to pre-fetch and cache all known CRLs. Not everyone is so lucky to know what those CRLs are and where to get them from (until the client arrives). OCSP requests are small, discreet transactions, usually on the order of 1-2K tops. An OCSP responder does the pre-fetching and caching, and the OCSP client simply makes a request for some specific piece of information. In that respect, OCSP can be highly performant. Plus it doesn't put any additional burden on the BIG-IP to store and manage large CRLs.
    • You have the same issue with OCSP that you'd have with CRLs if the remote service becomes unavailable. But then a local OCSP responder is easy to build, and many options are free. I worked for many years with US DoD and CRLs were never really an option for client cert revocation given the immense size of the CRLs. A local OCSP responder not only alleviates that, but you can also usually configure a responder to respond past the expiration of the CRL. That was sometimes a problem too, where the CRL would go stale before a new one could be fetched. With the local responder we could set it to return the last status of a cert in the previous copy of the CRL. I don't think that's possible with pure CRLs.

    As to the CRL refresh script option, the more I think about it, this probably wouldn't work in your scenario. The idea would be to have a script that perdiodically queries the site and passes a client cert, forcing the client SSL profile to fetch the CRL if it's not in cache. The issue here is that your script would need a client cert for each respective CRL. I imagine you could spoof this, but I'd have to test that to know for sure.

    • _JOHN_'s avatar
      _JOHN_
      Icon for Altocumulus rankAltocumulus
      Sep 15, 2022

      Cheers again for the reply 😊

      I agree in ‘normal’ operation OCSP has clear benefits over CRL.  However in my scenario I see clear advantages to pre-fetching the CRLs.  I can’t go into the finer specifics of my scenario, but I know it will apply to many others.  The key general concepts are:

      • Known CRLDPs (allows for pre-fetching on a regular basis, i.e. updated in background)
      • CRLs are small – they are many magnitudes smaller than the permissible sizes referenced by K10054 (even if concatenated into a single file – although I realise they will only grow over time, so concatenating gains more potential for issues over time)

      Unfortunately there doesn’t seem to be an option to pre-fetch the files natively in the F5 arsenal.  APM comes closest in that you can use an ‘Update Interval’ so as to continually refresh CRLs based on a timeframe of your choosing (unlike CRL Validator which relies on the nextupdate filed), but APM has downsides:

      • CRL only downloaded at point that client cert received (connections held up and as you say it’s for the whole CRL)
      • If multiple certs received at essentially the same time then I believe multiple CRL downloads are kicked off until one of them completes and the CRL is actually cached
      • I _believe_ (welcome to hear otherwise 😊 ) if the CRL download has been started based on the Update Interval then if that download fails the client connection is dropped – i.e. even though the CRL nextupdate hasn’t expired, the existing CRL is seen as invalid because the Update Interval has passed, so there is no kind of ‘fallback’ (whereas if CRL was pre-fetched in the background there would be no impact to current client connections if a CRL download fails – the system just waits for the next download to trigger and in the meantime the existing CRL is used).

      Regarding expired CRL – the SSL profiles allow for that option, but it is on a single CRL (unless trying to use concatenated CRLs).

      Local OCSP responder sounds interesting, but it will still hold up connections rather than just making a check against a local file.  A local OCSP connection should be very quick, but surely it would still be longer than a local compare on the F5 against a CRL?  You then also have to manage something separate to the F5, which is a headache for companies when they have bought the F5 to do all of this work (e.g. keeping on top of security issues and applying patches, more devices to bring under a support model and to manage alerts from etc. etc.).

      Regarding forcing a refresh by presenting various client certs (one per CRL) - having the certs to present would be no problem at all in my scenario – test certs are available and constantly refreshed.  However it wouldn’t really do what I need – I want the CRL to be refreshed in the background.  There is sufficient load that real traffic would likely trigger a CRL download long before the automated script did.

      Unfortunately there just doesn’t appear to be a method to do what I need (other than possibly concatenated CRL if we can get round the support issues) so I will look into the alternatives.  CRL Validator won’t be one of them based on the cache being for nextupdate (would be simply too long between refreshes).  APM is a potential but with those downsides I mentioned.

  • Kevin_Stewart's avatar
    Kevin_Stewart
    Icon for Employee rankEmployee
    Sep 15, 2022

    Just a few additional points, don't really want to belabor the OCSP topic.

    • OCSP will be faster in situations where there are large CRLs. An OCSP request is a tiny binary payload wrapped in a single HTTP request. The total transaction (req and rep) is usually less that 1K.
    • A local OCSP would need to know the remote CRLs it's managing so it can go get them and cache. That's no different than the CRL situation you have now. And since the responder knows what and where they are, it'll always have a copy (fresh or otherwise) of that CRL and can make a valid OCSP response.
    • My suggestion then, if local CRL validation is what you're aiming for, is to create a timed script that simply generates a query to the BIG-IP VIP and performs mTLS with a cert from each of the known issuers. This will force the BIG-IP to always have an up-to-date CRL in cache.
  • Martin_Ĺ ebek's avatar
    Martin_Ĺ ebek
    Icon for Altostratus rankAltostratus
    Feb 05, 2024

    I am also concerned about dynamic CRLs.  Anyone knows how to clear cached CRL file?  Let's assume BIG-IP cached the CRL with Next Update after one week.  What if I, for any reason, need to start from scratch and make BIG-IP to fetch the CRL again?  I have not found a way how to achieve this.

    Kevin Stewart mentioned "You could always run a script to periodically issue a request that triggers a CRL download", but I am not sure how to do it.  Even if I create a new CRL under cert-validator (/sys crypto cert-validator crl) and I use it in client-ssl profile, it uses previously cached CRL without tryint to fetch the CRL.

    -----------------------------------
Sys::CRL: XXXXXXXXX
-----------------------------------
Total Fetches                     0
Successful Fetches                0
New CRL Fetch Attempts            0
Cached CRL Updates                0
Internal Errors                   0
                                
Connection Errors               
  HTTP Errors                     0
  Timeouts                        0
  Other Failures                  0
                                
Validation Errors               
  Parsing Failures                0
  Verification Errors             0
  Validity Errors                 0
  Other Errors                    0
                                
Total Certificate Status Queries  2
                                
Certificate Status              
  Good                            2
  Revoked                         0
  Unknown                         0
  • F5-Enthusiast-X's avatar
    F5-Enthusiast-X
    Icon for Altocumulus rankAltocumulus
    Apr 11, 2024

    Hello together,

     

    interesting topic so far also for me.

    I found that that the dynamic CRL files seems to be cached really long. We have one dynamic CRL which got queried 6 times in about 6 month based on output of "tmsh show sys crypto cert-validator crl". 

     

    These dynamic CRLs seems to be saved under: "/config/filestore/crl_file_cache_d/"
    Unfortunately, these are not in standard openssl x509 CRL format.
    So I can't verify them with "openssl crl -in <crl-file-name> -noout -text". Result is currently:

    "unable to load CRL
    140058772170416:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:697:Expecting: X509 CRL"

     

    We would also like to know, how to verify the content of the current loaded CRL for troubleshooting purpose. Also it is needed for us to have a way of forcing a re-download a CRL.

     

    Regards