Forum Discussion

Sakiy's avatar
Sakiy
Icon for Altocumulus rankAltocumulus
Oct 21, 2024

LTM SSL handshake failuer (40) with IIS SSL setting Accept

I had an issue that communication from client PC failed with one of pool members. Clinet PC can directly access to the problem member without any issue. If it is accessed through VS, the failure happened.

 

As investigated with packet capture, following error caused the communication failure.

 

Transport Layer Security
    TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Handshake Failure)
        Content Type: Alert (21)
        Version: TLS 1.2 (0x0303)
        Length: 26
        Alert Message
            Level: Fatal (2)
            Description: Handshake Failure (40)

As I investigated, I found the problem member's IIS SSL setting is set as "Accept". Other working members are set as "Ignore". As I changed the setting to "Ignore", the problem was gone.

 

The IIS SSL setting "Accept" is to accept clinet certificate if it is provided by client. If client did not provide client cetificate, IIS still establish connection. On the VS, SSL server profile is used. the profile setting is almost default.

Do you know why BIG-IP fails the SSL communication if the IIS SSL setting is "Accept"?

 

LTM
ssl
  • BellaAbzug1's avatar
    BellaAbzug1
    Icon for Mist rankMist
    Oct 21, 2024

    An LTM SSL handshake failure (40) indicates a problem during the SSL/TLS handshake process between the Local Traffic Manager (LTM) and an IIS server. This issue often arises when there’s a mismatch in SSL settings, such as protocol versions or cipher suites. If the IIS SSL setting is set to Accept, it might be allowing weaker protocols, which could cause compatibility issues with LTM. To resolve this, ensure that both the LTM and IIS are configured to support the same SSL/TLS protocols and cipher suites. Additionally, checking the server certificates for validity and proper installation can help eliminate handshake failures.