Failure creating certificate acme challenge 404 error in BIG-IP F5 WAF

[Dineshan]

We have more than 600 government websites behind the BIG-IP system. We have done almost 60% of certificates created and offloaded. Suddenly we couldn't create any certificate and got the below error. This error not only for one website. Now we can't renew or create a new certificate.

We use fanceg/letsencrypt -in GitHub to integrates Let's Encrypt with BigIP ( GitHub - fanceg/letsencrypt-bigip).

INFO: Using main config file /etc/dehydrated/configProcessing verugal.ds.gov.lk

• Signing domains...
• Generating private key...
• Generating signing request...
• Requesting new certificate order from CA...
• Received 1 authorizations URLs from the CA
• Handling authorization for verugal.ds.gov.lk
• 1 pending challenge(s)
• Deploying challenge tokens...
• Responding to challenge for verugal.ds.gov.lk authorization...
• Cleaning challenge tokens...
• Challenge validation has failed : (
• ERROR: Challenge is invalid! (returned: invalid) (result: ["type"] "http-01"
• ["status"] "invalid"
• ["error","type"] "urn:ietf:params:acme:error:unauthorized"
• ["error","detail"] "Invalid response from http://verugal.ds.gov.lk/.well-known/acme-challenge/CnhSunPlqtFks1odEZVDOs_0OScqWBzf_xDejAo14WE 1 [43.224.124.166]: 404"
• ["error","status"] 403
• ["error"] {"type":"urn:ietf:params:acme:error:unauthorized","detail":"Invalid response from http://verugal.ds.gov.lk/.well-known/acme-challenge/CnhSunPlqtFks1odEZVDOs_0OScqWBzf_xDejAo14WE 1 [43.224.124.166]: 404","status":403}
• ["url"] "https://acme-v02.api.letsencrypt.org/acme/chall-v3/12995442889/eoq1dQ"
• ["token"] "CnhSunPlqtFks1odEZVDOs_0OScqWBzf_xDejAo14WE"
• ["validationRecord",0,"url"] "http://verugal.ds.gov.lk/.well-known/acme-challenge/CnhSunPlqtFks1odEZVDOs_0OScqWBzf_xDejAo14WE 1"
• ["validationRecord",0,"hostname"] "verugal.ds.gov.lk"
• ["validationRecord",0,"port"] "80"
• ["validationRecord",0,"addressesResolved",0] "43.224.124.166"
• ["validationRecord",0,"addressesResolved"] ["43.224.124.166"]
• ["validationRecord",0,"addressUsed"] "43.224.124.166"
• ["validationRecord",0] {"url":"http://verugal.ds.gov.lk/.well-known/acme-challenge/CnhSunPlqtFks1odEZVDOs_0OScqWBzf_xDejAo14WE","ho..."}
• ["validationRecord"] [{"url":"http://verugal.ds.gov.lk/.well-known/acme-challenge/CnhSunPlqtFks1odEZVDOs_0OScqWBzf_xDejAo14WE","ho..."}]
• ["validated"] "2021-05-10T04:46:36Z")
• Processing vrc.bopepoddala.ds.gov.lk
• Signing domains...
• Generating private key...
• Generating signing request...
• Requesting new certificate order from CA...
• Received 1 authorizations URLs from the CA
• Handling authorization for vrc.bopepoddala.ds.gov.lk
• 1 pending challenge(s)
• Deploying challenge tokens...
• Responding to challenge for vrc.bopepoddala.ds.gov.lk authorization...
• Cleaning challenge tokens...
• Challenge validation has failed : (
• ERROR: Challenge is invalid! (returned: invalid) (result: ["type"] "http-01"
• ["status"] "invalid"
• ["error","type"] "urn:ietf:params:acme:error:unauthorized"
• ["error","detail"] "Invalid response from http://vrc.bopepoddala.ds.gov.lk/.well-known/acme-challenge/v9qnpu7SA8F5xFQjk0VgltT__yVviLmyGftlvTAK... [43.224.124.166]: 404"
• ["error","status"] 403
• ["error"] {"type":"urn:ietf:params:acme:error:unauthorized","detail":"Invalid response from http://vrc.bopepoddala.ds.gov.lk/.well-known/acme-challenge/v9qnpu7SA8F5xFQjk0VgltT__yVviLmyGftlvTAK... [43.224.124.166]: 404","status":403}
• ["url"] "https://acme-v02.api.letsencrypt.org/acme/chall-v3/12995448812/pq_1KA"
• ["token"] "v9qnpu7SA8F5xFQjk0VgltT__yVviLmyGftlvTAKY9Y"
• ["validationRecord",0,"url"] "http://vrc.bopepoddala.ds.gov.lk/.well-known/acme-challenge/v9qnpu7SA8F5xFQjk0VgltT__yVviLmyGftlvTAK..."
• ["validationRecord",0,"hostname"] "vrc.bopepoddala.ds.gov.lk"
• ["validationRecord",0,"port"] "80"
• ["validationRecord",0,"addressesResolved",0] "43.224.124.166"
• ["validationRecord",0,"addressesResolved"] ["43.224.124.166"]
• ["validationRecord",0,"addressUsed"] "43.224.124.166"
• ["validationRecord",0] {"url":"http://vrc.bopepoddala.ds.gov.lk/.well-known/acme-challenge/v9qnpu7SA8F5xFQjk0VgltT__yVviLmyGftlvTAK..."}
• ["validationRecord"] [{"url":"http://vrc.bopepoddala.ds.gov.lk/.well-known/acme-challenge/v9qnpu7SA8F5xFQjk0VgltT__yVviLmyGftlvTAK..."}]
• ["validated"] "2021-05-10T04:46:58Z")

Can anyone help me out with this issue? Are there any process changes or updates in letsencrypt site or BIG-IP intigrations? Due to this lots of government websites affected!

[Nikoolayy1]

For such issues like "of government websites affected!" also raise a TAC case as this is a comunity website that does not have SLA.

404 error could be related to lest encrypt issue not F5, check the the forums:

https://community.letsencrypt.org/t/404-not-found-when-creating-certificate/93743

https://community.letsencrypt.org/t/cant-generate-cert-404-not-found/104184/2

Also lets encrypt has limits as it is a free service and maybe not the best for government websites:

https://letsencrypt.org/docs/rate-limits/