pool member sending Alert (Level: Fatal, Description: Unknown Certificate [46]) for a new SSL Cert of a VS

masajjad · ‎22-Jul-2020

Current wildcard cert (2048 bit key size) about to expired used for SSL offloading on HTTPS VS. New cert (4096 bit key size) was applied to the that VS and application breaks.

VS server also has SSL Server profile (factory default). The tcpdump shows F5 Self IP sends client hello. Pool member (Tomcat) sends certificate installed on backend to F5 Self IP. Next there is client key exchange from F5 Self IP. Followed by encrypted handshake message and application data exchange between them.

Next pool member sends client hello to VS. VS sends server hello and the new cert to pool member. But, pool member sends Alert (Level: Fatal, Description: Unknown Certificate [46]) to VS. And communication ends with TCP FIN.

Can't wrap my head around. Any suggestion?

Lidev · ‎23-Jul-2020

Hi masajjad,

if you use SSL 2 ways between the Virtual Server and the Tomcat server. It's necessary that the Tomcat server can check if the certificate is signed by an authorized CA.

Have you checked if the Tomcat has the CA who signed the certificate in his trust store?

Regards

masajjad · ‎23-Jul-2020

Appreciate your reply Lidev. Earlier same client side cert was installed on Tomcat server. Now that wildcard is expiring Security does not want us to install new client cert on back end. We installed a new cert on back-end that was signed by our "local CA". F5 knows that local CA. From tcpdump we see can see that new cert being used to secured communication between F5 (self IP) and pool member (only one server in the back-end). Afterwards we also see VS (not self IP) is also sending the new cert as part of SSL handshake that is where application breaks. Hope it shades some light. Thanks.

masajjad · ‎23-Jul-2020

I came across this https://mta.openssl.org/pipermail/openssl-users/2017-April/005683.html

> Secure Sockets Layer

> TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Certificate Unknown)

> Content Type: Alert (21)

> Version: TLS 1.2 (0x0303)

> Length: 2

> Alert Message

> Level: Fatal (2)

> Description: Certificate Unknown (46)

Client objects to the server chain. Either does not trust the MiTM root CA, or

is unhappy about its encoding (assuming tshark is not generating an FP warning).

And here is the tcpdump analysis that highlights our situation.

Bank-end sends Locally signed Cert to F5 Self IP. F5 likes it and we see Client Key Exchange.
VS sends new 4096 bit key size client cert (signed by Entrust that signs current one as well) to back-end
Back-end flags Alert Cert Unknown.

Lidev · ‎23-Jul-2020

Thank you for this clarification and the tcpdump.

Can you post you SSL profile configuration to help debugging ?

Also, disable Generic Alert in SSL profile, when the setting is disabled, the system sends more specific SSL alert messages.

masajjad · ‎23-Jul-2020

Thanks for you time and effort to help me out. Current wildcard is expiring this Sunday. So, it's a bit of crisis.

Disabling Generic Alert in client SSL profile did not add much detail. Both cases I see only handshake error message between pool member and VS. Nothing much besides this error.

I had tail -f /var/log/ltm running

Jul 23 06:32:49 Ottciof5101 warning tmm1[21959]: 01260013:4: SSL Handshake failed for TCP 10.5.15.120:62226 -> 10.5.29.11:443

Jul 23 06:38:11 Ottciof5101 warning tmm1[21959]: 01260013:4: SSL Handshake failed for TCP 10.5.15.120:62242 -> 10.5.29.11:443

This being a wildcard cert it was used on multiple client SSL profile. I created a custom parent SSL profile pointed the new cert there. And all old SSL profiles were extended from that custom parent instead of factory default

(/Common)(tmos)

ltm profile client-ssl clientssl_Wildcard_COMPANY {

app-service none

cert-key-chain {

Wildcard_COMPANY_2020_Entrust_Chain_Bundle_Dec_2030_0 {

cert Wildcard_COMPANY_2020.com

chain Entrust_Chain_Bundle_Dec_2030

key Wildcard_COMPANY_2020.com

}

defaults-from clientssl

inherit-ca-certkeychain true

inherit-certkeychain false

}

(cfg-sync In Sync)(Active)(/HR)(tmos)#

ltm profile client-ssl HR-Accero-SNI_cfmws.com-SHA2 {

app-service none

cert-key-chain {

Wildcard_COMPANY_2020_Entrust_Chain_Bundle_Dec_2030_0 {

cert /Common/Wildcard_COMPANY_2020.com

chain /Common/Entrust_Chain_Bundle_Dec_2030

key /Common/Wildcard_COMPANY_2020.com

}

defaults-from /Common/clientssl_Wildcard_COMPANY

generic-alert disabled

inherit-ca-certkeychain true

inherit-certkeychain true

server-name *.abc.com

sni-default true

}

ltm profile client-ssl HR-Accero-SNI_cfpsa.com_SHA2 {

app-service none

cert-key-chain {

Wildcard_COMPANY_2020_Entrust_Chain_Bundle_Dec_2030_0 {

cert /Common/Wildcard_COMPANY_2020.com

chain /Common/Entrust_Chain_Bundle_Dec_2030

key /Common/Wildcard_COMPANY_2020.com

}

defaults-from /Common/clientssl_Wildcard_COMPANY

generic-alert disabled

inherit-ca-certkeychain true

inherit-certkeychain true

server-name *.xyz.com

}

Lidev · ‎23-Jul-2020

The configuration of Client SSL profiles looks good.

Can you provide the result of theses commands please ?

openssl s_client -connect <virtual_server>:<port>
openssl s_client -connect <backend>:<port>

masajjad · ‎23-Jul-2020

Excuse my delay. Pardon me... instead of replying back I was writing answer. Organized mess 🙂

@syslog:~$ openssl s_client -connect 10.5.29.11:443

CONNECTED(00000003)

verify return:1

depth=0 C = XX, ST = XX, L = XX, O = X Y Z, XX = *.abc.com

verify return:1

---

Certificate chain

0 s:/C=xxxxx/CN=*.abc.com

---

Server certificate

Server public key is 4096 bit

Verify return code: 0 (ok)

syslog:~$ openssl s_client -connect 10.5.15.120:443

CONNECTED(00000003)

depth=0 C = XX, ST = XX, L = XX, O = xxxxxxx, OU = IT, CN = *.abc.com

verify error:num=20:unable to get local issuer certificate

verify return:1

depth=0 C = XX, ST = XX, L =XX, O = xxxxxxxx, OU = IT, CN = *.abc.com

verify error:num=21:unable to verify the first certificate

verify return:1

---

Certificate chain

0 s:/C=xx/ST=xx/L=xx/O=xxxxxxxx/OU=IT/CN=*.abc.com

i:/DC=com/DC=domain/CN=COLOCAL-CA

---

Server certificate

-----BEGIN CERTIFICATE-----

subject=/C=XXXXXXXX/OU=IT/CN=*.abc.com

issuer=/DC=com/DC=domain/CN=COLOCAL-CA

---

No client certificate CA names sent

Server public key is 4096 bit

Verify return code: 21 (unable to verify the first certificate)

---

closed

I am taking the client doing SSL connection request does not have the Local CA cert installed.

Lidev · ‎23-Jul-2020

if the test were performed from the BIG-IP, you find the root cause of your issue 😉

Double check the CA assign on your SSL Client and Server profile (you may have forgotten to include intermediate chain) .

if you perform TLS 2 ways, check also Trusted Certificate Authorities.

masajjad · ‎23-Jul-2020

Hi Lidev,

Appreciate your continued feedback.

Cert of "Local CA" that signed the cert for back-end was imported to F5 (from System > File Management > SSL Certificate List).
From tcpdump we see back-end send that newly generated cert signed by the Local CA and F5 does client exchange. This suggests F5 knows about the Local CA that signed the cert for back-end
Issue arises when F5 sends (Public CA signed) cert to back-end. We send public CA chain bundle for both and new cert from SSL profile.

Am I missing something from your reply?

Thanks again.

Lidev · ‎23-Jul-2020

Hello,

It looks like an SSL configuration problem on the backend server side, enable SSL log debug on your F5 BIG-IP .

modify /sys db log.ssl.level value Debug

don't forget to disable SSL debug logging after by typing the following command: (modify /sys db log.ssl.level value Warning)

Also start a SSL Dump to monitor all SSL trafifc (https://support.f5.com/csp/article/K10209)

With all this, you will have more information about SSL traffic and you may have more insight into the problem you are facing.

masajjad · ‎24-Jul-2020

Hi Lidev,

Turns out there is some authentication component of the application that requires the frontend cert. We copied the external cert to backend and issue was fixed. We were chasing out own tail.

Thanks a lot for your time and effort.

Lidev · ‎27-Jul-2020

Hi masajjad,

I'm glad to hear it. please don't forget to mark the answer as "the best" in order to pass your post as resolved.

Regards

DevCentral

pool member sending Alert (Level: Fatal, Description: Unknown Certificate [46]) for a new SSL Cert of a VS