Current wildcard cert (2048 bit key size) about to expired used for SSL offloading on HTTPS VS. New cert (4096 bit key size) was applied to the that VS and application breaks. VS server also has SSL Server profile (factory default). The tcpdump shows F5 Self IP sends client hello. Pool member (Tomcat) sends certificate installed on backend to F5 Self IP. Next there is client key exchange from F5 Self IP. Followed by encrypted handshake message and application data exchange between them. Next pool member sends client hello to VS. VS sends server hello and the new cert to pool member. But, pool member sends Alert (Level: Fatal, Description: Unknown Certificate [46]) to VS. And communication ends with TCP FIN. Can't wrap my head around. Any suggestion?

Hi masajjad, if you use SSL 2 ways between the Virtual Server and the Tomcat server. It's necessary that the Tomcat server can check if the certificate is signed by an authorized CA.Have you checked if the Tomcat has the CA who signed the certificate in his trust store? Regards

Thank you for this clarification and the tcpdump. Can you post you SSL profile configuration to help debugging ?Also, disable Generic Alert in SSL profile, when the setting is disabled, the system sends more specific SSL alert messages.

Hello,It looks like an SSL configuration problem on the backend server side, enable SSL log debug on your F5 BIG-IP .modify /sys db log.ssl.level value Debug don't forget to disable SSL debug logging after by typing the following command: (modify /sys db log.ssl.level value Warning)Also start a SSL Dump to monitor all SSL trafifc (https://support.f5.com/csp/article/K10209)With all this, you will have more information about SSL traffic and you may have more insight into the problem you are facing.

Appreciate your reply Lidev. Earlier same client side cert was installed on Tomcat server. Now that wildcard is expiring Security does not want us to install new client cert on back end. We installed a new cert on back-end that was signed by our "local CA". F5 knows that local CA. From tcpdump we see can see that new cert being used to secured communication between F5 (self IP) and pool member (only one server in the back-end). Afterwards we also see VS (not self IP) is also sending the new cert as part of SSL handshake that is where application breaks. Hope it shades some light. Thanks.

I came across this https://mta.openssl.org/pipermail/openssl-users/2017-April/005683.html > Secure Sockets Layer > TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Certificate Unknown) > Content Type: Alert (21) > Version: TLS 1.2 (0x0303) > Length: 2 > Alert Message > Level: Fatal (2) > Description: Certificate Unknown (46) Client objects to the server chain. Either does not trust the MiTM root CA, or is unhappy about its encoding (assuming tshark is not generating an FP warning). And here is the tcpdump analysis that highlights our situation.Bank-end sends Locally signed Cert to F5 Self IP. F5 likes it and we see Client Key Exchange.VS sends new 4096 bit key size client cert (signed by Entrust that signs current one as well) to back-endBack-end flags Alert Cert Unknown.

pool member sending Alert (Level: Fatal, Description: Unknown Certificate [46]) for a new SSL Cert of a VS

Lidev
MVP
Jul 23, 2020
Hi masajjad,

if you use SSL 2 ways between the Virtual Server and the Tomcat server. It's necessary that the Tomcat server can check if the certificate is signed by an authorized CA.
Have you checked if the Tomcat has the CA who signed the certificate in his trust store?

Regards
Lidev
MVP
Jul 23, 2020
Thank you for this clarification and the tcpdump.
Can you post you SSL profile configuration to help debugging ?
Also, disable Generic Alert in SSL profile, when the setting is disabled, the system sends more specific SSL alert messages.
masajjad
Cirrus
Jul 23, 2020
Appreciate your reply Lidev. Earlier same client side cert was installed on Tomcat server. Now that wildcard is expiring Security does not want us to install new client cert on back end. We installed a new cert on back-end that was signed by our "local CA". F5 knows that local CA. From tcpdump we see can see that new cert being used to secured communication between F5 (self IP) and pool member (only one server in the back-end). Afterwards we also see VS (not self IP) is also sending the new cert as part of SSL handshake that is where application breaks. Hope it shades some light. Thanks.
masajjad
Cirrus
Jul 23, 2020
I came across this https://mta.openssl.org/pipermail/openssl-users/2017-April/005683.html

> Secure Sockets Layer
> TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Certificate Unknown)
> Content Type: Alert (21)
> Version: TLS 1.2 (0x0303)
> Length: 2
> Alert Message
> Level: Fatal (2)
> Description: Certificate Unknown (46)

Client objects to the server chain. Either does not trust the MiTM root CA, or
is unhappy about its encoding (assuming tshark is not generating an FP warning).

And here is the tcpdump analysis that highlights our situation.
Bank-end sends Locally signed Cert to F5 Self IP. F5 likes it and we see Client Key Exchange.
VS sends new 4096 bit key size client cert (signed by Entrust that signs current one as well) to back-end
Back-end flags Alert Cert Unknown.
masajjad
Cirrus
Jul 23, 2020
Thanks for you time and effort to help me out. Current wildcard is expiring this Sunday. So, it's a bit of crisis.

Disabling Generic Alert in client SSL profile did not add much detail. Both cases I see only handshake error message between pool member and VS. Nothing much besides this error.

I had tail -f /var/log/ltm running

Jul 23 06:32:49 Ottciof5101 warning tmm1[21959]: 01260013:4: SSL Handshake failed for TCP 10.5.15.120:62226 -> 10.5.29.11:443

Jul 23 06:38:11 Ottciof5101 warning tmm1[21959]: 01260013:4: SSL Handshake failed for TCP 10.5.15.120:62242 -> 10.5.29.11:443

This being a wildcard cert it was used on multiple client SSL profile. I created a custom parent SSL profile pointed the new cert there. And all old SSL profiles were extended from that custom parent instead of factory default

(/Common)(tmos)

ltm profile client-ssl clientssl_Wildcard_COMPANY {
  app-service none
  cert-key-chain {
    Wildcard_COMPANY_2020_Entrust_Chain_Bundle_Dec_2030_0 {
      cert Wildcard_COMPANY_2020.com
      chain Entrust_Chain_Bundle_Dec_2030
      key Wildcard_COMPANY_2020.com
    }
  }
  defaults-from clientssl
  inherit-ca-certkeychain true
  inherit-certkeychain false
}

(cfg-sync In Sync)(Active)(/HR)(tmos)#

ltm profile client-ssl HR-Accero-SNI_cfmws.com-SHA2 {
  app-service none
  cert-key-chain {
    Wildcard_COMPANY_2020_Entrust_Chain_Bundle_Dec_2030_0 {
      cert /Common/Wildcard_COMPANY_2020.com
      chain /Common/Entrust_Chain_Bundle_Dec_2030
      key /Common/Wildcard_COMPANY_2020.com
    }
  }
  defaults-from /Common/clientssl_Wildcard_COMPANY
  generic-alert disabled
  inherit-ca-certkeychain true
  inherit-certkeychain true
  server-name *.abc.com
  sni-default true
}
ltm profile client-ssl HR-Accero-SNI_cfpsa.com_SHA2 {
  app-service none
  cert-key-chain {
    Wildcard_COMPANY_2020_Entrust_Chain_Bundle_Dec_2030_0 {
      cert /Common/Wildcard_COMPANY_2020.com
      chain /Common/Entrust_Chain_Bundle_Dec_2030
      key /Common/Wildcard_COMPANY_2020.com
    }
  }
  defaults-from /Common/clientssl_Wildcard_COMPANY
  generic-alert disabled
  inherit-ca-certkeychain true
  inherit-certkeychain true
  server-name *.xyz.com
}
Lidev
MVP
Jul 23, 2020
The configuration of Client SSL profiles looks good.
Can you provide the result of theses commands please ?
openssl s_client -connect <virtual_server>:<port> openssl s_client -connect <backend>:<port>
- masajjad
  Cirrus
  Jul 23, 2020
  Excuse my delay. Pardon me... instead of replying back I was writing answer. Organized mess :)
  
  @syslog:~$ openssl s_client -connect 10.5.29.11:443
  CONNECTED(00000003)
  depth=2 C = US, O = "Entrust, Inc.", OU = See www.entrust.net/legal-terms, OU = "(c) 2009 Entrust, Inc. - for authorized use only", CN = Entrust Root Certification Authority - G2
  verify return:1
  depth=1 C = US, O = "Entrust, Inc.", OU = See www.entrust.net/legal-terms, OU = "(c) 2012 Entrust, Inc. - for authorized use only", CN = Entrust Certification Authority - L1K
  verify return:1
  depth=0 C = XX, ST = XX, L = XX, O = X Y Z, XX = *.abc.com
  verify return:1
  ---
  Certificate chain
  0 s:/C=xxxxx/CN=*.abc.com
    i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
  1 s:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
    i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2009 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - G2
  2 s:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2009 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - G2
    i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2009 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - G2
  ---
  Server certificate
  
  Server public key is 4096 bit
  
    Verify return code: 0 (ok)
  
  syslog:~$ openssl s_client -connect 10.5.15.120:443
  CONNECTED(00000003)
  depth=0 C = XX, ST = XX, L = XX, O = xxxxxxx, OU = IT, CN = *.abc.com
  verify error:num=20:unable to get local issuer certificate
  verify return:1
  depth=0 C = XX, ST = XX, L =XX, O = xxxxxxxx, OU = IT, CN = *.abc.com
  verify error:num=21:unable to verify the first certificate
  verify return:1
  ---
  Certificate chain
  0 s:/C=xx/ST=xx/L=xx/O=xxxxxxxx/OU=IT/CN=*.abc.com
    i:/DC=com/DC=domain/CN=COLOCAL-CA
  ---
  Server certificate
  -----BEGIN CERTIFICATE-----
  
  subject=/C=XXXXXXXX/OU=IT/CN=*.abc.com
  issuer=/DC=com/DC=domain/CN=COLOCAL-CA
  ---
  No client certificate CA names sent
  
  Server public key is 4096 bit
  
    Verify return code: 21 (unable to verify the first certificate)
  ---
  closed
  
  I am taking the client doing SSL connection request does not have the Local CA cert installed.
  - Lidev
    MVP
    Jul 24, 2020
    Hello,
    It looks like an SSL configuration problem on the backend server side, enable SSL log debug on your F5 BIG-IP .
    modify /sys db log.ssl.level value Debug
    don't forget to disable SSL debug logging after by typing the following command: (modify /sys db log.ssl.level value Warning)
    Also start a SSL Dump to monitor all SSL trafifc (https://support.f5.com/csp/article/K10209)
    With all this, you will have more information about SSL traffic and you may have more insight into the problem you are facing.
Lidev
MVP
Jul 27, 2020
Hi masajjad,
I'm glad to hear it. please don't forget to mark the answer as "the best" in order to pass your post as resolved.

Regards